bbc/simorgh

Simorgh

BBC World Service News websites are rendered using Simorgh, a ReactJS based application. Simorgh also renders AMP news article pages for World Service and Public Service News.

Simorgh provides a fast and accessible web experience used by millions of people around the world each month (see list of websites using Simorgh). It is regularly maintained and well documented, and we welcome open source contributors.

Simorgh is primarily maintained by the BBC News Web Engineering teams. It delivers highly trusted news to readers all over the world, currently in (41 languages). We support a wide range of devices and care deeply about scale, performance, and accessibility. We work in agile, flexible teams, and have an exciting roadmap for future development.

Documentation index

Please familiarise yourself with our:

• Code of conduct
• Coding Standards
• Contributing guidelines
• Guide to Code Reviews
• GPG Signing Guide
• Primary README (you are here)
• Recommended Tools
• Troubleshooting

NB there is further documentation colocated with relevant code. The above list is an index of the top-level documentation of our repo.

Before Installation

Please read: CONTRIBUTING.md

Installation

Install Node. https://nodejs.org/en/. We use the version specified in .nvmrc and if you have a node version manager (nvm) you can run the following script to automatically change to the project supported version.

nvm use

Install Yarn

The Simorgh project uses Yarn for package management. It is recommended to install Yarn through the npm package manager, which comes bundled with Node.js when you install it on your system. To install Yarn, run this command:

npm install --global yarn

Then you can run the following commands to install Simorgh

git clone [email protected]:bbc/simorgh.git
cd simorgh
yarn install

Local Development

Export a relevant BFF_PATH so that simorgh can connect to the relevant fabl instance:

# For a local instance of fabl
export BFF_PATH="http://localhost:3210/module/simorgh-bff"

Start the application:

cd ws-nextjs-app
yarn dev

The nextJS application will start on http://localhost:7081.

Once the application is running, you can visit a valid route, e.g. http://localhost:7081/pidgin.

Secret Environment Variables

Some local-development commands used by BBC staff require additional environment variables that are not stored in this repository. BBC staff can follow this guide (requires access) to set them on your machine.

Feature toggles

Some features are hidden/shown based on feature toggles set remotely via BBC internal systems. By default when running the application locally however, these values are not fetched remotely and are set in these config files. As such, these may differ between environments. Please read the toggles documentation for more information and for instructions on how to fetch remote toggle values locally.

Article pages

Article pages are served at routes of the format /news/articles/:id where id is the asset ID generated by the Content Management System.

FYI: Article explaining the BBC's use of ids in URL

These two News articles are available on the Test environment of our CMS, as well as locally, so are often used for testing:

• http://localhost:7081/news/articles/c6v11qzyv8po
• http://localhost:7081/persian/articles/c4vlle3q337o.

We are also serving AMP HTML pages at the route /news/articles/:id.amp https://www.ampproject.org

• http://localhost:7081/news/articles/c6v11qzyv8po.amp
• http://localhost:7081/persian/articles/c4vlle3q337o.amp.

Services with variants can't be accessed using the format above, instead the variant must be provided in the URL.

• http://localhost:7081/zhongwen/articles/c3xd4x9prgyo/simp
• http://localhost:7081/zhongwen/articles/c3xd4x9prgyo/simp.amp.

Home pages

World Service home pages are served in the format /:service where service represents a World Service site:

• http://localhost:7081/igbo
• http://localhost:7081/pidgin

Services with variants can't be accessed using the format above, instead the variant must be provided in the URL.

• http://localhost:7081/zhongwen/simp
• http://localhost:7081/zhongwen/simp.amp.

World Service home pages do not support AMP.

Topic Pages

Topic pages use internal BBC APIs that are not publicly accessible. This can cause the following warnings to appear when developing locally:

No BFF_PATH set as environment variable, you will not have access to topics

Internal developers who need to work on topic pages locally should contact the team for access. See Secret Environment Variables for how to set the required environment variables.

Other page types

You can find other pages types by looking through our routes and their associates regexes, but we suggest you start with the above then have a look at the core of the application to understand and find the other routes.

See a list of local route examples here.

Storybook (UI Development Environment/Style Guide)

We use Storybook for developing components in isolation from the Simorgh Application. You can access this at https://bbc.github.io/simorgh/

To run locally yarn storybook, it will then be available at http://localhost:9001/. Introduction to and documentation for Storybook is here: https://storybook.js.org/basics/introduction/.

When viewing Video stories locally, make sure to use a BBC domain, as outlined in the changing request location section. Video will not work in the hosted version of Storybook linked above for this reason.

We also use Chromatic QA to run cross-browser testing on our stories.

Storybook compiles JavaScript, TypeScript and JSX with SWC, configured in .storybook/main.ts. It uses Emotion's JSX runtime and SWC plugin.

When updating Storybook's SWC compiler or @swc/plugin-emotion, check their plugin compatibility and validate Emotion-styled stories as well as SCSS Modules and MDX documentation.

Please also note that if you would like to see the components rendered with our fonts, you will need to force a repaint of the canvas. This is because our fonts all have the font-display property of optional or swap in accordance with the respective loading strategies here: https://ws-downloads.files.bbci.co.uk/fonts/index.html. The easiest way to force a repaint is just to move the divider between the preview window the and Knobs section or resize the browser window.

Configuring the application to run on a local network

If you want to host the application to be accessible through your local network, follow the instructions here.

Production build locally

To run this application locally with a production build, run: yarn build && yarn start.

We use yarn build locally which bundles the application pointing at localhost for data and static assets.

Using environment builds locally

This is mainly used for debugging latest using the TEST and LIVE environment bundles. Ensure that the bundles exist in the static asset location for the correct environment before starting to debug.

To run TEST bundles on localhost:

• In envConfig/test.env change the values of:
  • LOG_DIR='/var/log/simorgh' to LOG_DIR='log'
• Then run rm -rf build && yarn build:test && yarn start
• Visit a test article: http://localhost:7081/news/articles/c0g992jmmkko

To run LIVE bundles on localhost:

• In envConfig/live.env change the values of:
  • LOG_DIR='/var/log/simorgh' to LOG_DIR='log'
• Then run rm -rf build && yarn build:live && yarn start
• Visit a live article: http://localhost:7081/news/articles/c8xxl4l3dzeo

Changing request location

Some features perform differently dependant on whether a user is located within the UK or internationally. You can explicitly request a specific version by accessing Simorgh via a specific localhost BBC domain:

• UK version: http://localhost.bbc.co.uk:7081/news/articles/c0000000001o
• International version: http://localhost.bbc.com:7081/news/articles/c0000000001o

If these urls do not work, you may need to add a hosts file entry (/etc/hosts or C:\Windows\System32\drivers\etc\hosts):

127.0.0.1 localhost.bbc.co.uk
127.0.0.1 localhost.bbc.com

Production build on CI

On deployment make buildCi is run in the CI environment which creates bundles for both the test and live environments. On the two environments the .env.test or .env.live files overwrite the .env file which is used to run the application with the correct bundles.

Bundle analysis reports

Every run of yarn build will update the bundle analysis files in the repo. To view a breakdown of the bundle size, open the generated html report in a browser ./reports/webpackBundleReport.html This is generated via webpack-bundle-analyzer. The data is also available as json ./reports/webpackBundleReport.json.

Tests

Linting and unit tests

We have linting with the Airbnb styleguide and we use Prettier as a code formatter. They can be run with yarn test:lint.

ESLint parses JavaScript and JSX with its built-in parser and TypeScript and TSX with @typescript-eslint/parser.

Both yarn test:lint and yarn test:lint:fix use a content-based cache in .eslintcache to skip unchanged files on subsequent runs. The existing lint scope and rules are unchanged. Use yarn test:lint --no-cache for a full uncached run, for example after updating dependencies or Prettier configuration. ESLint's per-file cache does not track changes to imported files, so use an uncached run when checking cross-file changes.

We have Jest unit tests that can be ran with yarn test:unit for the root projects and yarn test from the ws-nextjs-app directory for the Next.js project.

The repository has one root jest.config.ts containing seven named projects: Unit Tests, Unit Tests (Client), Puppeteer Tests, Unit Tests (Next), Integration Tests - Canonical, Integration Tests - AMP and Integration Tests - Lite. The root and Next project groups use the same SWC compiler with separate project-group options and test environments. Root projects use SWC without app-derived compiler options; Next projects retain the app-derived settings from the Next workspace.

Root and Next unit and integration projects share src/testHelpers/setupTests.js for DOM matchers, Web APIs, crypto mocks, setImmediate and console handling. All these projects fail on unexpected console errors, with the existing warning suppressions retained. Browser-specific error filtering only runs when a window is available, so Node-only API tests use the same file. Location mocking is left to individual tests rather than replacing the environment's location globally. Root-only mocks and fixture environment values still run earlier through src/testHelpers/jest-setup.js; Next integration tests retain their real fetch. Puppeteer keeps its separate setup.

yarn test:unit selects the two root unit projects. The Next workspace's yarn test selects only Unit Tests (Next), and its integration scripts select their named platform projects. All workspace commands explicitly use the root configuration so running them from either repository directory resolves the same projects. The root integration command continues to manage the app lifecycle through ws-nextjs-app/integration/utils/runTests.ts.

Resolving the combined configuration initializes the Next app configuration and its .env files even when only root projects are selected. This is an intentional trade-off of keeping one configuration, so root test commands may observe app environment values during configuration resolution. A single global jest-junit reporter writes results to the repository root's reports/jest directory, regardless of the invocation directory.

SWC preserves read-only module exports. Tests that need to spy on or replace imported exports use explicit jest.mock factories to provide mockable exports. Existing implementations can be preserved with jest.requireActual and object spreading.

End-to-end tests

Main application

We use Cypress for our end-to-end tests. To run the smoke tests locally, run this single command:

yarn test:e2e

It will spin up a production server on port 7081 and run the Cypress tests against that. To run the smoke tests interactively, run:

yarn test:e2e:interactive

This loads a user interface which easily allows for individual tests to be run alongside a visual stream of the browser, as the tests run.

Environment variables

There are several environment variables you can use with our test suite, which are:

These commands can be run in combination.

Full suite of tests

The default way to run the e2e suite aka yarn test:e2e or yarn test:e2e:interactive runs a subset of our tests, otherwise know as smoke tests. To run the full suite:

CYPRESS_SMOKE=false yarn test:e2e

Limiting scope of runs

Tests can be restricted to only run for a single service by specifying it using the CYPRESS_ONLY_SERVICE environment variable. For example:

CYPRESS_ONLY_SERVICE=urdu yarn test:e2e

To run only a particular spec it is necessary to invoke Cypress directly. First ensure Simorgh is already running in another tab and then run (for example, to only run article tests):

npx cypress run --spec cypress/integration/pages/articles/index.js

Further details on using the Cypress CLI can be found at https://docs.cypress.io/guides/guides/command-line.html

Running e2e in the UK against LIVE

This affects developers based in the UK only (but may affect you if you're using a VPN routing through the UK)

Cypress .visit() function is locked to visiting a single domain per test. This becomes problematic when you launch the e2e tests from within the UK, due to redirects from .com to .co.uk. By default cypress tests will run as if they were ran outside of the uk. In order to run these tests from the UK you have to pass in the UK Cypress environment variable to the tests. This will replace the URL endings to .co.uk, which will allow you to run these tests successfully.

Here is an example command:

CYPRESS_APP_ENV=test CYPRESS_UK=true CYPRESS_SMOKE=true yarn cypress

Running e2e outside EU

This affects developers based out of the EU (but may affect you if you're using a VPN routing through a country not in the EU)

Running Cypress tests outside the EU will not show the EU consent banners on AMP, and this may cause some tests to fail. Set CYPRESS_SKIP_EU=true to prevent these tests from running when outside the EU.

An example command will be:

CYPRESS_SKIP_EU=true yarn cypress:interactive

The following command runs both simorgh and cypress:

CYPRESS_APP_ENV=local CYPRESS_UK=true CYPRESS_SMOKE=true yarn test:e2e

CYPRESS_APP_ENV can also be set equal to 'test' and 'live'. CYPRESS_SMOKE can be true or false. It is true by default and runs a specific subset of tests.

Lighthouse Best Practice tests

We use Lighthouse CI to test our pages. A single config, lighthouse/lighthouserc.js, is shared by local runs, GitHub Actions and our CodeBuild pipelines. LIGHTHOUSE_APP_ENV selects which URLs are audited and which thresholds apply.

To run it against a local Next.js server on port 7081:

yarn lighthouse

To run it against the test or live environments (as CodeBuild does):

LIGHTHOUSE_APP_ENV=test yarn lighthouse

Local reports are written to .lighthouseci/.

Why is it called Simorgh?!

Named Simorgh after the Persian mythological bird. The Simorgh is the amalgam of many birds (and in some accounts other animals) into one.

Happily, a metaphor which seemed apt for offering all BBC articles in one solution is perhaps now even more appropriate as the application evolves to support more content types. It’s also a clear reference to the international nature of our teams, but also to the desire to ensure articles (and everything which has followed) works for users in all languages the BBC supports.

It is also a unique name which is practical and, more superficially, the bird is very pretty.