AI safety by design for kids' learning apps, 2026

Safe AI for children is mostly a set of design choices: no open chat for young children, screens before and after every model call, nothing a child says or sounds like kept longer than needed, a budget on every call, and AI kept out of decisions about the child. The rules behind this tightened in 2025 and 2026, from the updated COPPA rule to California's companion-chatbot law. Peabody (peabody.kids), free through the end of 2026, publishes how it applies each one, and where it falls short.

What changed, rule by rule

COPPA, 2025. The US Federal Trade Commission finalised changes to the Children's Online Privacy Protection Rule in January 2025; they were published in April 2025 and companies had until 22 April 2026 to comply. Sharing a child's data with third parties now needs its own consent, separate from general consent. "Personal information" now includes biometric identifiers such as voiceprints and facial patterns. Data may be kept only as long as reasonably necessary, under a written retention policy, and companies need a written security programme. A proposed ban on engagement tricks like push notifications was dropped. Separately, the industry self-regulator CARU acted against a voice-based AI tutor for children in 2025 over notice and consent.

UK Children's Code. The Information Commissioner's Office has said that generative AI developers and deployers may fall in scope of the Children's Code where children are likely to use the system or its outputs.

EU AI Act. Annex III lists as high-risk any AI system intended to evaluate learning outcomes, including when used to steer the learning process, or to assess the level of education a person should receive. Article 5 prohibits emotion recognition in education except for medical or safety reasons. When the high-risk duties apply is still being reported differently by different sources, so check the Official Journal before relying on a date.

California SB 243. Signed in October 2025 and in effect from 1 January 2026, it requires operators of companion chatbots to disclose that the user is talking to an AI, to remind known minors repeatedly, to prevent sexual content for them, and to run a suicide and self-harm protocol, with a private right of action. New York passed a parallel law.

The model makers' rules. Anthropic, whose models many kids' apps use, requires organisations serving minors to put in place age verification, content moderation, monitoring and reporting, to tell users they are talking to AI, and to state publicly that they comply with laws like COPPA. OpenAI added principles for under-18 users to its Model Spec in December 2025.

The design patterns that follow

Hamsa Bastani and colleagues' 2024 study in Turkey is the clearest case for the second row: students given plain GPT-4 during practice did better in practice and then 17% worse on an exam without it, while a hint-only version mostly removed the harm. A 2025 design for children's AI tutors set out the layered approach: constraints in the prompt, input threat detection, an independent moderation agent, then human review.

How Peabody applies them

Peabody's full AI policy is on how Peabody uses AI. In short:

• No open chat with an AI, and none at all for young children. A child's story ideas go in once and a story comes out. In Peabody Studio a child's ask builds a game. The one conversation with a model is Talk with Peabody, a short typed chat for 3rd grade and up in a family account: code screens every message before a model sees it, every reply is checked twice before a child sees it, and the family's grown-ups can read it.
• Layered screens. Code refuses contact details, off-limits words and profanity in an idea; an AI classifier screens it, with a stronger model's second opinion before turning it away for a child in a family account. Every story is checked by code, then screened by AI as pass, flag or block with a scariness label, with a second opinion on anything short of a clear pass. Flagged stories are held for Aaron, who runs Peabody, to review.
• AI never decides about the child. Levels, the next question and reviews are ordinary code; a story's level is measured from its text.
• No emotion detection. Peabody records what a child answers and how long it took, and never tries to read how a child feels.
• A budget on every call. Each AI call names the child it is for, with default limits of $1 a month and $0.15 a day of AI per child, and a brake of 40 calls a day that never stops a safety screen. The ledger keeps no prompts, ideas or names.
• Names kept out where it can. The writer gets a family child's first name only while the grown-up leaves "put in stories" on; never an age, grade, school or email.
• An AI switch per child. A grown-up can turn Peabody's AI off for one child. Then nothing is written for that child by a model, and only the safety screens on what they read still run.

Where Peabody falls short

Publishing the gaps is part of the pattern.

• Speech is not always on the device. A microphone button turns speech into typed words. Peabody tries the browser's own recognizer first, which in browsers like Chrome sends the sound to the browser maker's service; only where that fails does Peabody's on-device speech model take over. Peabody itself never receives or stores audio.
• Typed ideas are sent as typed. Contact details are refused, but a name or school a child types into an idea is not removed for family children, and Studio asks are not stripped of names.
• The model maker's terms are not yet on the privacy page. Anthropic's data-use and retention terms for Peabody's account have still to be confirmed and published.

What this does not settle

• Laws differ by place, and this post is not legal advice. Peabody is offered in the US today.
• No filter is perfect. Layers reduce misses; they do not remove them.
• Safety is not learning. A safe app can still be a poor teacher. Peabody has not been tested in a trial.

For the ten questions to ask any app, see is AI safe for kids' learning?. Start free or read a story from the library.

Learn more elsewhere

• UNICEF guidance on AI and children: the global benchmark for child-centred AI, revised in December 2025 with new sections on companions.
• UNICEF: when AI becomes a friend: a 2026 brief on companion chatbots and how six places regulate them.
• 5Rights children and AI design code: a framework for designing AI that respects children.
• ICO: the Children's Code: what a well-built kids' service should do by default.

For about seventy more, checked and sorted by who they're for, see where to learn about AI.

Questions parents ask

Is AI safe for kids?

It depends on the design. The safer pattern is no open chat for young children, screens before and after every model call, nothing about the child decided by a model, speech kept on the device, and a public page that says all of it, gaps included.

What did the 2025 COPPA changes do?

They require separate parental consent before a child's data is shared with third parties, count biometrics such as voiceprints as personal information, limit how long data is kept under a written policy, and require a written security programme. Companies had until 22 April 2026 to comply.

Can a kids' app use AI to decide a child's level?

In the EU, AI that evaluates learning or steers it is listed as high-risk, with heavy duties attached. Peabody keeps levels, the next question and reviews in ordinary code, and uses AI only to write and screen.

Does Peabody record my child's voice?

No. Nothing is recorded or sent to Peabody. When a child uses a microphone button to dictate, the browser's own recognizer, tried first, may send the sound to the browser maker; the fallback runs on the device.