thunlp/UltraChat

We report three evaluations in this section: Alpaca-Eval from Stanford, Evol-instruct from Microsoft's WizardLM, and our curated evaluation set. Evaluations of modern LLMs may be biased and affected by many factors, we are also actively working on more comprehensive evaluation methods.

Alpaca-Eval

AlpacaEval is a leaderboard specifically designed for evaluating LLMs. The leaderboard is created based on the win-rate against Text-Davince-003 automatically evaluated by GPT-4.

Evol-instruct

This dataset is constructed with an evolutionary strategy by rewriting the instructions through multiple rounds to obtain instructions at different complexity levels. The benchmark is developed by the WizardLM project, another excellent chat language model!

Our Evaluation Set

We curate an evaluation set, encompassing the Vicuna Benchmark and additional 300 questions and instructions generated by GPT-4. The questions/instructions cover a wide range of topics, including commonsense, world knowledge, professional knowledge (specifically physics and biology) , mathematics, and writing tasks on different levels of difficulty. We use GPT-4 for evaluation. Here is the dataset.

• Example A: Question of Common Knowledge

  User: Which animal is known as man's best friend?

  UltraLM: There are many animals that humans enjoy as companion animals, but one of the absolute standout answers to the question "which animal is known as a man's best friend?" is the domestically-bred dog. Domesticated dogs have been by humans' sides for tens of thousands of years, providing faithful companionship and help; guarding people and property; and lending emotional support in everything from long, lonely days to military deployments. Dogs come in an almost mind-boggling variety of breeds of various sizes, with different energy levels, personalities, and temperaments, meaning there's almost certainly a pup out there who will be a perfect fit for whoever needs it most.

• Example B: Commonsense Reasoning

  User: Which weighs more: a pound of feathers or a pound of bricks?

  UltraLM: A pound of feathers and a pound of bricks both weigh exactly one pound. This is because one pound is a unit of weight, just as a foot is a unit of length or a second is a unit of time. We use the pound (or other unit of weight) to describe the weight of different objects, regardless of what that object is made of. So the answer to your question is that a pound of feathers and a pound of bricks both weigh the same amount.

• Example C: Reasoning

  User: If a farmer has 17 sheep and all but 9 die, how many are left?

  UltraLM: The total number of sheep that are left is 9.

• Example D: Reasoning

  User: If a red house is made from red bricks, and a blue house is made from blue bricks, what is a greenhouse made from?

  UltraLM: A greenhouse is not made from green bricks, but rather from materials such as glass or plastic that allow light to pass through and create a controlled environment for plants to grow. Greenhouses can come in a variety of colors, including green, but the color of the building itself does not affect the plants growing inside.

• Example E: Creation

  User: Write a poem about electromagnetism.

  UltraLM: Electromagnetism is a force we can't ignore,
  It's the reason for many things we adore. In a magnetic field, a wire will glow,
  And the electrons will align, both old and new.
  Electricity and magnetism are hand in hand,
  One creates a field, the other has a brand.
  The magnetic field can push and pull,
  On a wire with current, it will react in full.
  The attraction and repulsion it brings,
  Are the reason for many things.
  From motors to generators, and much more in between,
  Electromagnetism is at the root of what it means.
  It's a force that's all around us,
  From the compass to the light switch.
  With its power, it can do much,
  From a simple needle to a rocket's thrust.
  It's a force we can't deny,
  For it's a part of our lives.
  Electromagnetism is something to admire,
  For its strength and its might, it has no equal, it's sure.

• Example F: Creation

  User: Write a short story about the first second of the universe.

  UltraLM: In the first second of the universe, darkness reigned supreme. There was no light, no sound, no shape, no substance. Nothing yet was, but there was an intense power, a raw energy waiting to be unleashed. Within infinity, the first second held the potential for creation in its grasp. And then, with a mighty bang, a bright point of light emerged into existence. In it, the most fundamental particles were born, their forms yet unknown as they frenetically collided in a chaotic dance of creation. Within this primordial light, every force of nature and every component of life was stirred from pure energy to physical reality. The first atoms quickly assembled, and from these, stars and galaxies began to form. Their fusion creating a hot environment that pushed back the very void, shaping the cosmos through expanding and cooling waves. In this cosmic first, no structure of experience, not even time or space, could establish its rule. For an instant, the entirety of the universe existed in an embryonic nexus, a cosmic singularity from which all future existence originated. It was a microcosm forging the macrocosm of which it ultimately becomes a part, its moment of birth a metaphor for the cycles of creation that permeate the multiverse(s).From that single point of light, an infinite universe unfolded. For centuries, eons ahead, and forevermore, it remained the most foundational blink of its unceasing journey into the depths of the Great Beyond. The first second is the beginning of everything.

• 🌏 Questions about the World: The dialogue data in this sector is derived from a wide range of inquiries related to concepts, entities, and objects from the real world. The topics covered are extensive, spanning areas such as technology, art, and entrepreneurship.
• ✍🏻 Writing and Creation: The dialogue data in this sector is driven by the demands for writing/creation from scratch, and encompasses any tasks that an AI assistant may aid within the creative process, spanning from email composition to crafting narratives and plays, and beyond.
• 📋 Assistance on Existent Materials: The dialogue data in this sector is generated based on existing materials, including but not limited to rewriting, continuation, summarization, and inference, covering a diverse range of topics.

Meta Topics & Sub-Topics

• The data is derived from 30 representative and diverse meta topics (icons are from flaticon)

• Based on the above meta topics, we generate 1100+ subtopics for data construction
• For each subtopic, we generate up to 10 specific questions.
• Then we use Turbo APIs to generate new relevant questions for each of the 10 questions. We use hand-crafted prompts to instruct the model to generate a diverse set of questions covering a wide range of common concepts and objects.
• For each question, we generate a 3~7-round dialogue using the two models iteratively as described above.

Common Real-world Entities

• We gather top-frequent 10000 named entities from Wikidata.
• We generate 5 meta questions for each entity using ChatGPT API.
• For each meta question, we generate 10 more specific questions and 20 related but general questions.
• We sample 200k specific questions and 250k general questions along with the 50k meta-questions, and we generate a 3~7-round dialogue for each.

• We first collect 20 types of writing, as shown below (icons are from flaticon).

• For each type of writing, generate 200 different instructions that ask an AI assistant to generate text material, and 80% of the instructions are further expanded and detailed.
• Use the generated instructions as initial input and generate a 2~4-round dialogue each.

• We extract ~10w diverse materials from C4 dataset.
• We generate up to 5 questions/instructions for each piece of material.
• We combine the material with each question/instruction with a set of manually designed template as the initial input of a user to start a dialogue with AI assistant.
• For each input, we generate a 2~4-round dialogue.