When Software Becomes Labor: Who's Accountable When AI Agents Act on Our Behalf?

Amazon is suing Perplexity over something that sounds technical but raises profound questions about how we organize economic relationships. Perplexity's AI agent browses Amazon's store, evaluates products, and makes purchase recommendations—all on behalf of users who simply asked for help finding something to buy.

Perplexity's defense argues we should treat software as labor. That seemingly narrow legal strategy opens up questions that extend far beyond one lawsuit: When AI agents act autonomously on our behalf, who's actually responsible for what they do? And how do we design systems where meaningful accountability is even possible?

The Agency Problem

The traditional boundaries were clear. Humans made decisions, software executed instructions. But when an AI agent browses retail sites autonomously, evaluates options, and makes recommendations, it's performing tasks we'd call "work" if a human assistant did them. The legal question becomes: does that make it labor, or is it just sophisticated automation of the user's intent?

When you ask a human shopping assistant to find you a laptop, they're clearly acting on your behalf—your agent, your liability, your decision. But when Perplexity's AI does the same thing, the lines blur. Is Amazon dealing with you as a customer, or with Perplexity as an intermediary? Who's responsible if the recommendation is terrible or the transaction goes wrong?

If we accept "software as labor," what does that mean for accountability? A human employee has legal obligations, can be held liable, operates under labor laws. Software has... terms of service?

The clean principle would be: if a user tells an agent to do something, they should be accountable for its actions. Software companies bearing full liability for every autonomous action would either kill the technology before it starts or require explicit human approval for every action, defeating the purpose.

But there's a massive gap between what users think they're authorizing and what agents might actually do.

What users think they're authorizing: "Find me a good laptop under $1000"

What the agent might actually do: Scrape dozens of retail sites, execute search queries that trigger rate limiting, interpret "good" based on training data that might not match the user's values, make tradeoffs between price/performance/brand that embed assumptions the user never considered, potentially violate terms of service on multiple platforms.

This is the informed consent problem that's basically impossible to solve through disclosures. No one's going to read a 40-page explanation before asking an AI to find them a laptop. Even if they did, most users lack the technical knowledge to understand the implications.

We've seen this movie before with Terms of Service. Everyone clicks "I agree" without reading because actual informed consent would make the service unusable. But at least with ToS, you're just agreeing to a contract. With autonomous agents, you're potentially authorizing actions with legal or financial consequences you didn't anticipate.

A Framework for Risk

Not all agent actions carry the same stakes. Maybe accountability should be proportional to the level of autonomy granted and the consequences involved.

Low stakes, narrow scope: "Find me restaurant recommendations" → User accepts whatever the AI does, minimal accountability issues. If the agent shows you a bad restaurant, the consequences are trivial. You just don't go.

Medium stakes, bounded actions: "Compare laptops on Amazon and Best Buy" → User should understand the agent will access specific sites in specific ways. These might require explicit authorization per action or per session, with clear audit trails.

High stakes, broad autonomy: "Manage my investment portfolio" or "Handle my job applications" → This probably requires explicit authorization of specific actions, not just general intent. Your AI agent shouldn't be able to sign contracts or make medical decisions just because you told it to "handle things."

Reversibility adds another dimension. Actions that can be easily undone like cancelled orders and deleted messages are fundamentally different from ones that can't be reversed: signed contracts, published information, shared data. The harder something is to undo, the more friction we probably need before authorizing the agent to do it.

But here's where this gets complicated: risk to whom?

The user might be making a low-risk decision from their perspective while creating medium or high-risk conditions for others. One user's AI agent browsing Amazon is trivial. A million AI agents simultaneously crawling and transacting could fundamentally alter how the platform operates: load balancing, fraud detection, recommendation algorithms, pricing strategies all break down.

The Disintermediation Question

Nilay Patel at The Verge has been arguing that consumer companies are going to be disintermediated from customers by an agent layer. Amazon spent 30 years building a moat around the customer relationship, and now they're watching AI agents potentially turn them into a commodity backend.

The actions that are lowest-risk for individual users might be highest-risk for the platforms they're acting on. Amazon was designed around human behavior patterns like browsing speeds, attention spans, decision-making friction. When you remove that friction at scale, the system dynamics change completely.

This creates a coordination problem that's almost impossible to solve through individual user accountability. Each user is making a rational, low-risk decision to use an agent. But the collective effect is potentially destabilizing to the platforms they're using.

The disintermediation risk isn't evenly distributed:

High vulnerability: Retailers, comparison services, review sites, travel booking. Basically anywhere the main value is aggregation and discovery. If agents do that aggregation, the platform becomes just inventory.

Medium vulnerability: Services where there's switching cost or network effects, but agents could still change how people interact with them.

Low vulnerability: Businesses where the product is the relationship, like consulting, creative services, healthcare, or where trust and brand matter more than features.

Designing for a Mixed World

This isn't a binary future where either humans or agents control everything. The more interesting question is: what's the optimal human-to-agent ratio for different contexts?

For healthcare, you probably want a very low agent ratio. Clinical interactions are high-stakes, require nuanced human judgment, involve significant emotional labor, and have strong regulatory requirements. An agent might handle appointment scheduling, but the core interactions need to remain human-centered.

For financial planning, you might see a higher agent presence. The agent handles data gathering, runs scenarios, monitors accounts, flags anomalies. But the actual planning decisions—risk tolerance, life goals, tradeoffs between saving and spending—those probably stay human, at least for high-stakes choices.

For commodity retail, the ratio could flip entirely. If your value proposition is selection and convenience rather than experience or expertise, agents might become the primary interface.

This creates a design challenge that's fundamentally different from traditional user segmentation. With human users, you're designing for different needs and contexts, but they're all operating with similar constraints of attention, time, and cognitive load. With agents, you're designing for entities with radically different capabilities and limitations.

Agents are excellent at processing structured data at scale, executing multi-step procedures consistently, and operating without fatigue. They're terrible at interpreting ambiguous requirements, handling novel situations outside their training data, understanding emotional context, and taking accountability for mistakes.

Instead of optimizing for "ease of use," we might be optimizing for what I'd call clarity of intent transfer. This means making it as easy as possible for agents to accurately understand what humans actually want, while making it clear to humans what the agent can and can't reliably do.

Staying Human-Centered

How do you maintain human-centered design when the immediate "user" of your product might be an AI, but the ultimate beneficiary is still a human?

I think we still need to stay human-centered. The humans still matter most, even if they have agents working on their behalf. However, human needs will evolve and expand. Humans are always looking for the path of least resistance, so agents doing work for us is enticing. We have to manage this carefully because at the extreme, there could be problematic outcomes.

The strategic questions become:

For businesses: What ratio of human-to-agent interaction aligns with your value proposition and competitive positioning? Are you the brand humans trust for important decisions, or the efficient backend that agents use for transactions?

For designers: How do you structure APIs, data models, and interaction patterns to serve both human and agent users well? What needs to be agent-friendly (structured, programmatic, consistent) versus human-friendly (flexible, visual, forgiving)?

For organizations: Where do you want to preserve human friction, even if agents could remove it? What decisions are too important or too contextual to delegate?

For measurement: How do you track success when you can't directly observe user satisfaction because the user is an agent? If an agent completes a task "successfully" but the human who sent it isn't satisfied with the outcome, whose failure is that?

The Conversation We Need

The Perplexity-Amazon lawsuit will probably settle on narrow technical grounds about terms of service and web scraping. But the conceptual questions it raises (at what point does software become something more than a tool, and who's accountable when it acts autonomously) those will keep surfacing across domains.

This is genuinely uncharted territory. The technology is moving faster than our frameworks for thinking about accountability, design, and human agency. We're going to need new mental models for how to build products when the "user" might be an AI acting on behalf of a human we never directly interact with.

These aren't problems with obvious solutions. They're questions that different industries, products, and contexts will answer differently based on their specific constraints and values. The goal right now shouldn't be to prescribe answers, but to make sure we're asking the right questions.

What's the optimal human-to-agent ratio for your product or industry? Where should accountability sit when agents act autonomously? How do you design for clarity of intent when the interface is an AI rather than a human? What human friction is actually valuable to preserve?

These are conversations worth having now, before the patterns get locked in by legal precedent, business models, and user expectations that become hard to change.

What are you seeing in your industry or organization? Are agents already changing how people interact with your products? I'd be interested in hearing how different contexts are navigating these questions.