Will AI Make Us More Productive and Help Us Learn Faster?

This Week’s Focus: How AI Will Shape Work and Learning

AI is set to transform the workforce across a broad range of tasks, from manual roles like warehouse picking to specialized fields such as consulting. But will AI displace workers, or enhance human capabilities? This week, we explore these questions and examine how AI is influencing productivity, learning, and skill development in both low- and high-skilled domains. Considering recent studies, we look at how AI integration can either support or diminish human skills, sparking broader questions about the future of work across various sectors.

Over the last few months, I haven’t attended a single conversation in which people haven’t asked whether I see, or anticipate seeing, the impact of AI and LLM on work and productivity.

Not even one!

Needless to say, this is not a YES or NO response, and the answer keeps evolving. So, every several weeks, through this outlet, I’ll be sharing my observations, which are always backed by research and sprinkled with my boldly stated, yet weakly supported opinions.

It’s clear that AI is poised to change the nature of work across a wide spectrum of tasks, and understanding how it will impact productivity and learning across this spectrum is crucial. For example, will AI simply displace human workers, or will it complement and improve their abilities? 

In today’s article, we examine how AI influences three domains: low-skilled or manual work (warehouse picking), high-skilled or specialized work (consulting), and the educational processes that prepare individuals for either role.

As I delve into specific research findings, a clearer picture emerges of how AI can both improve and erode human capabilities, depending on how it is integrated into work processes. This outline sets the stage for broader questions about productivity, learning, and how AI will shape the future of work across different sectors.

 AI in Low-Skilled Work: Warehouse Picking and Manual Labor

Let’s start with the lower end of the skills spectrum.

The paper “AI-enabled Technology and Gig Workforce: The Role of Experience, Skill Level, and Task Complexity” studies whether AI complements or substitutes human skills in the gig economy, and how task complexity influences human-AI collaboration.

The gig economy is characterized by workers of varying levels of skill and experience. Most gig economy firms don’t require any type of evaluation during their hiring process, and instead, allow market dynamics to do their job through ranking and algorithmic pricing. AI tools designed to improve task efficiency, such as route optimization in grocery delivery or item-picking, promise to level the playing field between workers of varying abilities, but the question remains: Does AI complement or replace human experience?

Benjamin Knight (Instacart), Dmitry Mitrofanov (Boston College and a collaborator on another project), and Serguei Netessine (a Wharton colleague) conducted a large-scale field experiment on Instacart’s platform, introducing AI tools that helped workers optimize their in-store item-picking routes. The study assessed how AI affected performance based on workers’ skill levels, experience, and the complexity of their tasks​.

The study tracked the performance of gig workers over several months, focusing on metrics such as picking time (a measure of efficiency), refund rates (a measure of accuracy), and workload. To examine how AI influenced different cohorts, workers were divided into groups based on skill level and experience.

The main results are pretty interesting and are summarized nicely in the following diagrams:

AI as a Complement to Experience: More experienced workers saw significant performance improvements, especially on complex tasks where AI helped optimize their routes. AI did not diminish the value of their experience but instead enhanced it. These workers benefited from the synergy between their tacit knowledge and AI’s algorithmic precision​.

Challenges for Less Experienced Workers: Less experienced workers who relied more heavily on AI saw only modest gains. In some cases, their performance worsened as they struggled to fully integrate AI into their workflow, highlighting that experience remains crucial even when AI tools are available​.

Implications: AI can potentially complement human expertise, particularly in complex environments where task optimization is critical. However, it’s not a universal substitute for experience. This reinforces the idea that human experience remains vital to productivity, even in an AI-enhanced world.

In the realm of manual labor, AI has been surprisingly effective at complementing human work as it enhances productivity by performing computationally heavy tasks (e.g., route optimization or inventory management), allowing workers to focus on execution. In high-complexity environments—where tasks like picking items in a crowded warehouse demand both speed and accuracy—AI acts as a valuable tool, significantly boosting workers’ efficiency without replacing them.

However, the study also shows what AI can’t currently do: AI struggles to fully substitute the physical and intuitive skills humans bring to the table in manual labor. For example, while AI can suggest optimal routes for picking items, it cannot adapt in real-time to unforeseen issues like sudden congestion, misplaced items, or damaged products. Experienced workers bring tacit knowledge—knowing how to navigate a specific warehouse, which items tend to go out of stock, or how to multitask when things go wrong—that AI currently cannot replicate​.

We can use this study to identify potential pitfalls. The most significant risk here is AI’s uneven impact on workers of varying skill levels. In this case, AI could exacerbate existing inequalities in the workforce, as experienced workers could improve their productivity while newer, less skilled workers will struggle to keep up. This would create a situation where AI enhances the best workers but leaves others behind, potentially leading to job displacement for those who cannot adapt.

AI in High-Skilled Work: Consulting and Knowledge-Intensive Roles

Let’s look at the other end of the spectrum.

The paper “Leveraging Artificial Intelligence for Inclusive Service Operations: Empirical Evidence from a Professional Service Platform“ studies whether AI can improve decision-making processes by reducing biases and increasing inclusivity in service operations jobs.

Human decision-making is often influenced by unconscious biases, such as favoring responses from people with more authority or adopting neutral tones in open-ended tasks. This bias can undermine the diversity of perspectives, especially in service operations. With its capacity to analyze large amounts of text, AI could potentially enhance inclusivity by reducing the impact of these biases.

In collaboration with a large professional service platform in the U.S., Bowen Lou (USC), Kejia Hu (University of Oxford and a former student of mine), and Bilal Baloch (Enquire AI) studied the impact of introducing an AI-enabled text-processing tool into the response approval process. The AI was designed to assist human agents in approving answers by ranking responses based on inclusivity and relevance​.

The study was conducted on a professional service platform that connects an extensive network of experts to answer complex, open-ended customer questions across various fields such as finance, education, and technology. The platform’s operations rely on human agents to review and approve responses submitted by service providers with diverse expertise and backgrounds. This setup allowed the researchers to observe how AI could potentially reduce human biases in the approval process, particularly for neutral tones and authoritative figures, thus limiting the diversity of perspectives represented in the answers approved for customers.

Without prior announcement, the platform introduced an AI-enabled text processing tool into the answer review workflow. Based on a supervised neural network model, the AI system was designed to pre-screen and score responses based on relevance and inclusivity criteria. This scoring was intended to assist human agents in selecting answers that would better satisfy diverse customer inquiries. The AI model processed historical text-based interactions, analyzing question-answer pairs and using customer feedback to measure answer quality. The AI’s output included scores and rankings for each response, which human agents then used to guide their approval decisions.

This is how the flow looked like (post AI introduction):

Implementing AI created a quasi-experimental setting by providing an exogenous shock to the answer approval process. This allowed the researchers to analyze empirically how human agents interacted with AI rankings. 

The paper’s two most striking results are:

Mitigating Bias: AI successfully mitigated biases typically exhibited by human agents, such as the tendency to favor answers from more authoritative figures or those with neutral tones. AI-generated rankings led to more inclusive decisions, and this effect was particularly strong for less experienced agents.

Dependence on AI Rankings: Interestingly, human agents increasingly relied on AI rankings when making their decisions. This was particularly noticeable among less experienced agents, who seemed to defer to the AI when uncertain. While this reduced bias, it raised concerns about whether agents developed the critical skills needed for decision-making without AI assistance​.

Implications: AI can enhance decision-making by improving inclusivity and helping less experienced workers make better decisions. However, this raises a significant concern: if workers become overly reliant on AI rankings, they may fail to develop the tacit knowledge and critical thinking skills necessary for independent decision-making. This echoes the broader concern that while AI can optimize short-term performance, it may erode essential human capabilities over time.

AI in Education: Learning and Skill Development

So it’s clear that AI can help those with experience.

But what about those who met AI early in their development?

The paper “Generative AI Can Harm Learning” studies how human learning, particularly in educational settings, is affected.

As we all realized over the last two years, Generative AI tools, such as OpenAI’s GPT-4, can accelerate tasks by providing immediate answers and assisting with problem-solving. However, AI may inadvertently reduce human effort by making problem-solving easier and limiting the deeper cognitive engagement necessary for long-term learning. This is especially critical in domains where human oversight remains essential, as it risks diminishing human expertise.

Hamsa Bastani, Osbert Bastani, and Alp Sungu (all Penn and Wharton colleagues, and generally very nice people) and their co-authors conducted a randomized controlled trial involving nearly 1,000 students across several grades at a high school in Turkey. The students were split into three groups: one with access to a standard GPT-4 chat interface (GPT Base), another with access to a version that includes learning safeguards (GPT Tutor), and a control group with no AI access. The goal was to observe how AI impacts performance on practice problems and subsequent learning as measured by exams.

The experiment focused on high school math. Students completed a series of practice problems using either GPT-4 or traditional resources. The practice performance was measured against a final exam that assessed the same concepts but without any AI assistance​.

The results are extremely interesting:

Short-Term Gains: Both AI-assisted groups outperformed the control group during practice sessions. GPT Base users saw a 48% improvement in problem-solving performance, while GPT Tutor users saw a 127% improvement​. This confirms that AI can significantly boost short-term productivity —we could also just call it cheating as most students just asked ChatGPT for the answer. 

Long-Term Consequences: When AI was removed during the exam, students who had used GPT Base performed 17% worse than those in the control group, indicating that reliance on AI during practice diminished their in-depth learning. Interestingly, students using GPT Tutor, which provided hints instead of solutions, didn’t exhibit this drop in performance; their exam results were similar to those of the control group.

Implications: This study illustrates a critical trade-off: While AI can enhance short-term productivity by offering immediate solutions, it may harm deeper cognitive engagement necessary for skill retention. The use of AI in education must be carefully structured to avoid crutches that undermine learning. The difference between GPT Base and GPT Tutor highlights that AI can support learning if designed with safeguards that force students to engage more deeply with the material. The challenge lies in developing AI systems that strike this balance.

AI has the potential to revolutionize education by offering personalized tutoring, immediate feedback, and access to a vast repository of knowledge. AI can make education more accessible, providing assistance tailored to individual learning styles and helping students grasp complex concepts faster than traditional methods might allow.

The primary risk in education is that AI, while improving short-term outcomes, could erode the foundational skills critical for both high- and low-skilled work. If students become accustomed to using AI for problem-solving, they may struggle to adapt to tasks requiring independent thinking and long-term knowledge retention. This is especially concerning when preparing individuals for the workforce, where adaptability and critical thinking are essential. 

By acting as a crutch, AI could create a generation of workers who excel at using tools but lack the deeper understanding and expertise to navigate complex problems without them. This issue is particularly pronounced in education, as it directly affects the pipeline of talent feeding into both white- and blue-collar jobs.

Broader Implications: Human Adaptation and the Role of Tacit Knowledge

A clear pattern emerges across these studies: AI can significantly enhance productivity and learning in the short term, but its long-term benefits depend on how well humans adapt to working alongside these systems. 

Key takeaways:

Tacit Knowledge and Cognitive Engagement: The unspoken understanding gained through experience, remains irreplaceable in many tasks. In both educational and professional settings, overreliance on AI can erode this knowledge, leading to diminished capabilities when AI is not available. This is especially true in high-complexity environments where AI alone cannot capture the nuanced judgment required.

The Shortcut Problem: Humans naturally seek shortcuts, and AI offers powerful ones. However, as seen in the educational study, shortcuts provided by AI can inhibit the cognitive effort necessary for deep learning. It shouldn’t be that AI simply offers the answer; it must guide users through the discovery process, fostering engagement rather than dependence​. 

Changing How We Teach and Test: Integrating AI will require rethinking educational practices. Traditional tests emphasizing rote learning will become less relevant as AI becomes more pervasive. Instead, we must focus on teaching critical thinking and problem-solving skills that AI cannot automate while developing assessment methods that measure these abilities.

We will need to question what skills we want students to master and how we test their mastery of them. 

Of course, this is not new and is not only related to AI, but will be accelerated with AI.

For example, during my military days in an elite unit, one of the primary skills we practiced was navigation and orienteering. A typical exercise would have us plan routes, memorize them, and then navigate from memory throughout the night without a map or GPS. As it was the early days of GPS, devices were only used in exceptional cases (for actual operations), and mobile phones didn’t yet have built-in GPS. This often meant long nights, lost in the middle of nowhere until you finally honed the skill of navigation. 

You may be wondering whether these capabilities are still helpful today. Probably not for most people, unless you’re running the Barkley Marathons or operating in areas near war zones where GPS accuracy has been disrupted.

I’m not sure if soldiers these days can carry smartphones on these navigation exercises (which we spent weeks and weeks practicing), but if they are, it’s easy to see how, in the short run, it makes navigation significantly easier—perhaps even reducing it to an obsolete skill. In the long run, however, it only makes this essential skill harder to master, and this will be tested only at the most critical moments, when training may seem like a luxury. 

The Broader Risks of AI and LLMs: Displacement vs. Complementarity

As AI systems like LLMs become more powerful, the tension between improvement and displacement will only intensify. 

AI’s ability to enhance productivity is undeniable, but its potential to displace workers—both in white and blue-collar jobs—must be carefully managed. 

In consulting and other knowledge-intensive fields, AI can help workers reach higher productivity levels by automating routine tasks. However, the risk of losing critical human skills is real if workers become overly dependent on AI’s outputs. In manual labor, AI complements human experience, but again, the challenge lies in ensuring that all workers, regardless of skill level, can adapt to AI’s presence without being displaced.

Education is the canary in the coal mine for these changes. Here, we see the first signs of how AI can erode long-term learning and critical thinking while enhancing productivity in the short term. The education system must adapt to prepare future workers who are proficient in using AI and equipped with the tacit knowledge, creativity, and adaptability that AI cannot replicate.

Steve Jobs once compared computers to “bicycles for the mind.” LLMs take that concept and put it on steroids. But like any supplements, if given to people at the wrong stage of their development, they won’t build the right muscles.

If we rely on AI without simultaneously building complementary skills, we risk being replaced by it. 

Or, in the immortal words of Billy Beane: 

“Do you see what happens to the runt of the litter? 

He dies.”

I’m not a Luddite (as every Luddite says). I use AI and LLMs daily. But I encountered them later in my developmental stage. Do I believe AI should be banned or prohibited? No. But if we aim to test specific skills, we must ensure they can’t be “hacked” using AI. Guardrails, especially in early developmental stages, are essential to foster genuine growth alongside AI.