chef/supermarket

Supermarket

Supermarket is Chef's community repository for cookbooks, currently hosted at supermarket.chef.io. Supermarket can also run internally, behind-the-firewall.

The code is designed to be easy for others to contribute. The goal of the README file is to familiarize you with the project. You can find detailed information about Supermarket at Wiki page.

If you want to contribute to Supermarket, read the contributor's workflow for license information and other helpful tips that aid you in getting started. There are project artifacts such as planning docs, wireframes, recorded demos, and team retrospectives at public Google Drive folder.

If you have questions, feature ideas, or other suggestions, please open a GitHub Issue.

This repository has the code for the Supermarket application and the omnibus definition used to build the deb/rpm packages. Other Supermarket related repositories are:

• chef-cookbooks/supermarket-omnibus-cookbook: This cookbook is used to deploy Supermarket through the Supermarket omnibus package. For details on using this cookbook to install Supermarket omnibus, check out this webinar by the Supermarket Engineering team.

Development

Configuration

Configure the dotenv keys and secrets. See .env.example for required keys and secrets. docs/CONFIGURING.md page details the not-so-straightforward configuration information required to setup Supermarket working locally.

Local Environment

These instructions are tested and verified on macOS Catalina (10.15).

Dependency Services

As Docker Containers

1. Install docker

   brew cask install docker
   

Ensure you have a PostgreSQL version installed on the local filesystem for development libraries to be available for building the pg gem. See the instructions for locally running PostgreSQL below, however omit the steps where the service is running.

1. Start the docker containers.

   cd src/supermarket
   docker-compose up
   

As Locally Running Processes

1. Install Postgres by following any of the below instructions:

   • Install the Postgres App:

     This is probably the simplest way to get Postgres running on your mac. You can then start a Postgres server through the GUI of the app. If you go this route, then you'll have to add /Applications/Postgres.app/Contents/Versions/9.4/bin/ or the equivalent path to obtain the pg gem to build.

   • Through Homebrew:

     brew install postgresql
     

   When installed through homebrew, Postgres often requires additional configuration, see this blog post for instructions. Ensure to start the Postgresql server by following the command given below.

   pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
   

2. Install Redis. You can install it with Homebrew. Follow the instructions in the install output to start the redis server.

   brew install redis
   

   Run the redis server using command:

   redis-server --daemonize yes
   

Development Environment

1. Ensure you have Xcode installed.

2. Install a Ruby manager. If you don't already have one, you will need a Ruby manager to install the appropriate Ruby release, such as:

   • RVM
   • Rbenv
   • chruby
   • or any other Ruby version manager that may come along.

3. Use your ruby manager to install the necessary Ruby release. For instructions on this, please see the manager's documentation.

4. Ensure you have the Supermarket repo cloned to your machine. If not, clone it.

5. Navigate to that directory.

   cd <supermarket-repo>
   

6. Again, navigate to the src folder.

   cd src/supermarket
   

7. Install Bundler gem.

   gem install bundler:2.1.4 --user-install
   

8. Install required gems:

   bundle install
   

   Note: You might encounter the following errors. The possible fixes are also provided here.

   • Error installing gem: mimemagic
   • Fix-> brew install shared-mime-info
   • If you get error installing therubyracer and libv8 on macOS, follow the steps outlined in this link: https://billykong.github.io/ruby/2020/03/17/fixing-libv8-in-osx-catalina.html

9. Create the database, migrate to it, and then seed the database:

   bundle exec rails db:setup
   

10. Add required Postgres extensions.

    psql supermarket_development -c 'create extension plpgsql'
    psql supermarket_development -c 'create extension pg_trgm'
    

    Note: Ignore if the above 2 commands displays the following error: extension already exists.

11. Start the server:

    bundle exec foreman start
    

    Note: If you receive errors, ensure that redis and Postgres are running.

Setting up Auth

Supermarket uses oc-id running on a Chef Infra Server to authenticate users to Supermarket.

IF YOU ARE AN INTERNAL CHEF STAFFER - We will be performing some settings prior to set you up with oc-id. Consult the internal wiki on setting up your Supermarket dev environment (or ask a friendly team member!).

Note: Authentication currently requires a live Chef Infra Server running oc-id. We are working on a solution that would allow a developer to run the authentication locally. Stay tuned.

Create a new application and register it on oc-id (I called my application Application:Supermarket Development). Set the callback URL to http://localhost:3000/auth/chef_oauth2/callback or whatever localhost domain you use.

In your local copy of the Supermarket repo, copy the .env file to .env.development. Open up .env.development and replace these values:

CHEF_OAUTH2_APP_ID=YOUR_CHEF_OAUTH2_APP_ID
CHEF_OAUTH2_SECRET=YOUR_CHEF_OAUTH2_SECRET

with these values:

CHEF_OAUTH2_APP_ID=[Application ID of the oc-id application you just registered]
CHEF_OAUTH2_SECRET=[Secret of the oc-id application you just registered]

Restart your foreman server.

Now, when you click Sign In, you will be taken to your supermarket account with your Chef account!

Note: If you receive an omniauth csrf detected error, try clearing your browser's cache.

SPDX license linking for cookbooks

If a cookbook that has licenseId, which is also mentioned in SPDX license listed at https://github.com/spdx/license-list-data/blob/master/json/licenses.json, you need to update with respective license URL.

When a new cookbook is uploaded, license url information is fetched from the above link and is updated.

For an existing implementation of supermarket, there is a provision running with some ctl commands, for updating the information for cookbooks already there in system.

Following commands are available to update the license URLs for cookbooks:

Update licenses URLs for all the cookbooks in the system

spdx-all

Update for a single cookbook

spdx-latest <cookbook_name>

Update for a particular version of a cookbook

spdx-on-version <cookbook_name> <cookbook_version>

Connecting your GitHub Account

There are a couple of features that depend on GitHub integration (some quality metrics in Fieri) with your development environment. Follow these steps to create an application with your Github account:

1. Log into your Github account.
2. Click on your username in the upper right-hand corner. Your Profile page appears.
3. Click the Edit Profile button in the upper right corner of the Profile page.
4. Click Applications from the vertical menu on the left-hand side.
5. Click the Register new Application button from the section labeled Developer applications at the top of the screen.
6. Specify the name of your application (For example, Chef-Supermarket-Testing), then set the homepage URL as http://localhost:3000 (or whatever localhost domain that you use).
7. Set the Authorization callback URL to http://localhost:3000 (or your localhost domain of choice).
8. Click the Register application button.
9. Open the .env.development file in your local copy of the Supermarket repo.
10. Replace these values:

GITHUB_KEY=YOUR_GITHUB_KEY
GITHUB_SECRET=YOUR_GITHUB_SECRET

with:

GITHUB_KEY=[Your new application's client ID]
GITHUB_SECRET=[Your new application's client secret]

1. Next, create a Github Access token. You can also do this from the Applications section of your Profile page.

2. Navigate to the Personal access tokens section.

3. Click the Generate new token button.

4. When prompted, enter your Github password.

5. Enter the Token description. For example, testing-supermarket.

6. Leave the scopes at the defaults.

7. Click the Generate token button.

8. Copy the token generated and secure it safe!.

9. Open up your .env.development file again.

10. Replace this value:

GITHUB_ACCESS_TOKEN=YOUR_GITHUB_ACCESS_TOKEN

with:

GITHUB_ACCESS_TOKEN=[Token you just generated through Github]

Connecting your GitHub Enterprise Account

There are a couple of features that depend on GitHub Enterprise integration (some quality metrics in Fieri) with your development environment. Follow these steps to create an application with your Github account::

1. Log into your Github Enterprise account.
2. Click on your username in the upper right-hand corner.
3. Click User settings in the vertical menu on the right corner. Your public Profile page appears.
4. Click Developer settings in the vertical menu on the left-hand side.
5. Click the section labeled Developer settings with text OAuth App at the top of the screen.
6. Click new OAuth App. You can now register to a new OAuth application page.
7. Specify the name of your application. (For example, testing-supermarket-app), then set the homepage URL as http://localhost:3000 (or whatever localhost domain that you use).
8. Set the Authorization callback URL to http://localhost:3000/auth/github/callback (or your localhost domain of choice).
9. Click the **"Register application button.
10. Open up the .env.development file in your local copy of the Supermarket repo.
11. Replace these values:

GITHUB_KEY=YOUR_GITHUB_KEY
GITHUB_SECRET=YOUR_GITHUB_SECRET
GITHUB_ENTERPRISE_URL=YOUR_GITHUB_ENTERPRISE_URL
GITHUB_CLIENT_OPTION_SITE=YOUR_GITHUB_ENTERPRISE_SITE
GITHUB_CLIENT_OPTION_AUTHORIZE_URL=YOUR_GITHUB_ENTERPRISE_AUTHORIZE_URL
GITHUB_CLIENT_OPTION_ACCESS_TOKEN_URL=YOUR_GITHUB_ENTERPRISE_ACCESS_TOKEN_URL

with:

GITHUB_KEY=[Your new application's client ID]
GITHUB_SECRET=[Your new application's client secret]
GITHUB_ENTERPRISE_URL=[Your GitHub Enterprise URL]
GITHUB_CLIENT_OPTION_SITE=YOURGITHUBENTERPRISEURL/api/v3
GITHUB_CLIENT_OPTION_AUTHORIZE_URL=YOURGITHUBENTERPRISEURL/login/oauth/authorize
GITHUB_CLIENT_OPTION_ACCESS_TOKEN_URL=YOURGITHUBENTERPRISEURL/login/oauth/access_token

1. Next, create a GitHub Access token. You also do this from the Developer settings section.

2. Click Personal access tokens. The Personal access tokens page appears.

3. Navigate to the Personal access tokens section.

4. Click the Generate new token button.

5. When prompted, enter your GitHub password.

6. Enter the Token description. For example, testing-supermarket.

7. Leave the scopes at the defaults.

8. Click the Generate token button.

9. Copy the token generated and secure it safe!.

10. Open up your .env.development file again.

11. Replace this value:

GITHUB_ACCESS_TOKEN=YOUR_GITHUB_ACCESS_TOKEN

with:

GITHUB_ACCESS_TOKEN=[Token you just generated through Github]

Tests

Requirements for tests: Chromium-browser / Google-chrome, Node.

Run the entire test suite (rspec, rubocop and mocha) with:

bundle exec rake spec:all

Acceptance Tests

Acceptance tests are run with Capybara. Run rake spec:features to run the specs in spec/features. The default rake spec also runs these.

When writing feature specs, the Rack::Test driver is used by default. If the cuprite driver is required to be used (for example, an acceptance test that uses AJAX), add the use_cuprite: true metadata to the spec.

Some specs run using chromium-browser/Google Chrome, which must be installed for the test suite to pass.

JavaScript Tests

The JavaScript specs are run with Karma and use the Mocha test framework and the Chai Assertion Library.

The specs live in spec/javascripts. Run rake spec:javascripts to run the specs, and rake spec:javascripts:watch to run them continuously and watch for changes.

Node.js is required to run the JavaScript tests.

Background Jobs

Read about Supermarket's background jobs in the wiki.

Feature Flags

Supermarket uses a .env file to configure itself. Inside this file are key/value pairs. These key/value pairs will be exported as environment variables when the app runs, and Supermarket looks for these keys as environment variables when it needs to read a value that's configurable.

One of these keys is called FEATURES and it controls a number of features that can be turned on and off. Here are the available features that can be toggled:

• tools
• fieri
• announcement
• github
• no_crawl