Apple brings its GenAI ‘Apple Intelligence’ to developers, will let Siri control apps

Comment
Apple Intelligence, Apple’s new generative AI offering, won’t just be a consumer-facing feature; developers will be able to take advantage of the latest technology, too. In Apple’s keynote address at WWDC 2024 on Monday, the company announced that developers would be able to integrate the experience powered by Apple Intelligence into their own apps.
Apple’s SDKs (software development kits) have been updated with a variety of new APIs and frameworks that will allow app makers to do things like integrate the Image Playground — or GenAI image creation — with just a few lines of code. Apple showed off how an app like Craft could use this offering to make users’ documents more visual by allowing them to add AI images.
AI-powered writing tools will also be automatically available in any app that uses the standard editable text view. For this, Apple demonstrated how an app like Bear Notes would automatically be able to allow users to rewrite, proofread and summarize their notes.
In addition, Apple is building more ways for developers to take actions in apps with Siri.
Developers who have already adopted SiriKit — an SDK for integrating Siri into their apps — will see immediate enhancements for many of Siri’s new capabilities without any extra work on their part, Apple said. This includes areas like Lists, Notes, Media, Messaging, Payments, Restaurant reservations, VoIP calling and Workouts.
In its Developer keynote, Apple said that there are two new Siri capabilities that developers will be able to benefit from without additional work. First, Siri will be able to invoke any item from an app’s menus. That means a user could say something like “show my presenter notes” when in their slide deck or even something more conversational, like “I need to see my speaker notes.”
Secondly, Siri will be able to access any text displayed on the page using Apple’s standard text systems. This will allow users to reference and act on text on the screen. For instance, if you had a reminder or note to “wish grandpa a happy birthday,” you could just say “FaceTime him” to take action on that note.
Meanwhile, Apple’s App Intents framework, which allows for lightweight app-like interactions without the app being installed, will also gain access to Apple Intelligence.
Apple is defining new intents and making them available to developers across categories starting with a subset of categories including Books, Browsers, Cameras, Document Readers, File Management, Journals, Mail, Photos, Presentations, Spreadsheets, Whiteboards and Word Processors.
These intents are defined and tested so they’re easier for developers to adopt, Apple claims.
With the intents, a photo-editing app like Darkroom could leverage the Apply Filter intent so users could just say “Apply a cinematic present to the photo I took of Ian yesterday” to have the app take action. More domains will be added in time.
Initially, users will be able to develop with the Shortcuts app but over time, Siri will gain the ability to call the app intents in the supported domains.
Plus, Apple shared in its keynote address, apps that fit an existing SiriKit domain will be able to benefit from Siri’s enhanced conversational capabilities, like responding correctly even if you stumble over your words or understanding references to an earlier part of the conversation.
Siri will also be able to search data from apps using a new Spotlight API that enables app entities to be included in its index. These entities refer to Apple Intelligence’s semantic index of things like photos, messages, files, calendar events and more.
Also on Monday, the company announced its own password-manager app, AI-generated Bitmoji and Calculator for the iPad.
This post was updated after publication with more information from the Developer keynote.
Every weekday and Sunday, you can get the best of TechCrunch’s coverage.
Startups are the core of TechCrunch, so get our best coverage delivered weekly.
The latest Fintech news and analysis, delivered every Tuesday.
TechCrunch Mobility is your destination for transportation news and insight.
By submitting your email, you agree to our Terms and Privacy Notice.
The TechCrunch team runs down all of the biggest news from the Apple WWDC 2024 keynote in an easy-to-skim digest.
Hello and welcome back to TechCrunch Space. What a week! In the same seven-day period, we watched Boeing’s Starliner launch astronauts to space for the first time, and then we…
Elon Musk’s posts seem to misunderstand the relationship Apple announced with OpenAI at WWDC 2024.
“We’re looking forward to doing integrations with other models, including Google Gemini, for instance, in the future,” Federighi said during WWDC 2024.
When Urvashi Barooah applied to MBA programs in 2015, she focused her applications around her dream of becoming a venture capitalist. She got rejected from every school, and was told…
Slack CEO Denise Dresser is speaking at TechCrunch Disrupt 2024.
Apple kicked off its weeklong Worldwide Developers Conference (WWDC 2024) event today with the customary keynote at 1 p.m. ET/10 a.m. PT. The presentation focused on the company’s software offerings…
Apple’s SDKs (software development kits) have been updated with a variety of new APIs and frameworks.
Older iPhones or iPhone 15 users won’t be able to use these features.
Soon, Siri will be able to tap ChatGPT for “expertise” where it might be helpful, Apple says.
Apple Intelligence will have an understanding of who you’re talking with in a messaging conversation.
To use InSight, Apple TV+ subscribers can swipe down on their remote to bring up a display with actor names and character information in real time.
Siri is now more natural, more relevant and more personal — and it has new look.
The company has been pushing the feature as integral to all of its various operating system offerings, including iOS, macOS and the latest, VisionOS.
In addition to all the features you can find in the Passwords menu today, there’s a new column on the left that lets you more easily navigate your password collection.
With Smart Script, Apple says it’s making handwriting your notes even smoother and straighter.
iOS’ perennial tips calculating app is finally coming to the larger screen.
The new OS, announced at WWDC 2024, will allow users to mirror their iPhone screen directly on their Mac and even control it.
At Apple’s WWDC 2024, the company announced MacOS Sequoia.
“Messages via Satellite,” announced at Apple’s WWDC 2024 keynote, works much like the SOS feature does.
Apple says the new design will lead to less time searching for photos.
Users will be able to lock an app when they hand over their phone.
Apple’s WWDC 2024 keynote was packed, including a number of key new updates for iOS 18. One of the more interesting additions is Tap to Cash, which is more or…
In iOS 18, Apple will now support long-requested functionality, like the ability to set app icons and widgets wherever you want.
As expected, this is a pivotal moment for the mobile platform as iOS 18 is going to focus on artificial intelligence.
Apple today kicked off what it promised would be a packed WWDC 2024 with a handful of visionOS announcements. At the top of the list is the ability to turn…
The Apple Vision Pro is now available in eight new countries.
VisionOS 2 will come to Vision Pro as a free update later this year.
The security firm said the attacks targeting Snowflake customers is “ongoing,” suggesting the number of affected companies may rise.
French startup Kelvin, which uses computer vision and machine learning to make it easier to audit homes for energy efficiency, has raised $5.1M.
Powered by WordPress VIP

source
Sponsor:News technical sponsor
Sponsor:News AI sponsor
Sponsor: AI sponsor
Sponsor: AI sponsor