Posts Tagged ‘Python’

Watch the video

Introduction

Thierry Cordier and Valentin Laurent introduce MAPIE, a Python library within scikit-learn-contrib, designed for uncertainty quantification in machine learning models.

MAPIE on GitHub

Managing Uncertainty in Machine Learning

In AI applications — from autonomous vehicles to medical diagnostics — understanding prediction uncertainty is crucial. MAPIE uses conformal prediction methods to generate prediction intervals with controlled confidence, ensuring safer and more interpretable AI systems.

Key Features

MAPIE supports regression, classification, time series forecasting, and complex tasks like multi-label classification and semantic segmentation. It integrates seamlessly with scikit-learn, TensorFlow, PyTorch, and custom models.

Real-World Use Cases

By generating calibrated prediction intervals, MAPIE enables selective classification, robust decision-making under uncertainty, and provides statistical guarantees critical for safety-critical AI systems.

Conclusion

MAPIE empowers data scientists to quantify uncertainty elegantly, bridging the gap between predictive power and real-world reliability.

Exploring Quarto Dashboard for Impactful and Visual Communication

Watch the video

Introduction

Christophe Dervieux introduces us to Quarto Dashboard, a powerful open-source scientific and technical publishing system. Designed to create impactful visual communication directly from Jupyter Notebooks, Quarto enables the seamless creation of interactive charts, dashboards, and dynamic narratives.

Building Visual Communication with Quarto

Quarto extends standard markdown with advanced features tailored for scientific writing. It offers support for multiple computation engines, allowing narratives and executable code to merge into various outputs: PDF, HTML pages, websites, books, and especially dashboards. The dashboard format enhances data communication by organizing visual metrics in an efficient and impactful layout.

Using Quarto, rendering a Jupyter notebook becomes simple: with just a command-line instruction (quarto render), users can output polished, shareable dashboards. Additional extensions, such as those available in VS Code, JupyterLab, and Positron IDEs, streamline this experience further.

Dashboard Features and Design

Dashboards in Quarto organize content using components like cards, rows, columns, sidebars, and tabs. Each element structures visual outputs like plots, tables, and value boxes, allowing maximum clarity. Customization is straightforward, leveraging YAML configuration and Bootstrap-based theming. Users can create multi-page navigation, interactivity through JavaScript libraries, and adapt layouts for specific audiences.

Recent updates even enable branding dashboards easily with SCSS themes, making Quarto ideal for both scientific and corporate environments.

Conclusion

Quarto revolutionizes technical communication by enabling scientists and analysts to produce professional-grade dashboards and publications effortlessly. Christophe’s session at PyData Paris 2023 showcased the simplicity, power, and flexibility Quarto brings to modern data storytelling.

At PyConUS2024, Reuven M. Lerner, an esteemed independent trainer and consultant from Lerner Consulting, presented an exhaustive tutorial titled “All About Decorators.” This session aimed to strip away the perceived complexity surrounding Python’s decorators, revealing their inherent power and versatility. Reuven’s approach was to guide attendees through the fundamental principles, practical applications, and advanced techniques of decorators, empowering developers to leverage this elegant feature for cleaner, more maintainable code. The tutorial offered a deep dive into what decorators are, their internal mechanics, how to construct them, and when to employ them effectively in various programming scenarios.

Functions as First-Class Citizens: The Foundation of Decorators

At the heart of Python’s decorator mechanism lies the concept of functions as first-class objects. Reuven Lerner began by elucidating this foundational principle, demonstrating how functions in Python are not merely blocks of code but entities that can be assigned to variables, passed as arguments to other functions, and returned as values from functions. This flexibility is pivotal, as it allows for the dynamic manipulation and extension of code behavior without altering the original function definition.

He illustrated this with simple examples, such as wrapping print statements with additional lines of text. Initially, this might involve manually calling a “wrapper” function that takes another function as an argument. This manual wrapping, while functional, quickly becomes cumbersome when applied repeatedly across numerous functions. Reuven showed how this initial approach, though verbose, laid the groundwork for understanding the more sophisticated decorator syntax. The ability to treat functions like any other data type in Python empowers developers to create highly modular and adaptable code structures, a cornerstone for building robust and scalable applications.

The Power of Closures: Functions Returning Functions

Building upon the concept of first-class functions, Reuven delved into the powerful notion of closures. A closure is a function that remembers the environment in which it was created, even after the outer function has finished executing. This is achieved when an inner function is defined within an outer function, and the outer function returns this inner function. The inner function retains access to the outer function’s local variables, forming a “closure” over that environment.

Lerner’s explanations made it clear that closures are a critical stepping stone to understanding how decorators work. The decorator pattern fundamentally relies on an outer function (the decorator) that takes a function as input, defines an inner “wrapper” function, and then returns this wrapper. This wrapper function “closes over” the original function and any variables from the decorator’s scope, allowing it to execute the original function while adding pre- or post-processing logic. This concept is essential for functions that need to maintain state or access context from their creation environment, paving the way for more sophisticated decorator implementations.

Implementing the Decorator Pattern Manually

Before introducing Python’s syntactic sugar for decorators, Reuven walked attendees through the manual implementation of the decorator pattern. This hands-on exercise was crucial for demystifying the @ syntax and showing precisely what happens under the hood. The manual approach involves explicitly defining a “decorator function” that accepts another function (the “decorated function”) as an argument. Inside the decorator function, a new “wrapper function” is defined. This wrapper function contains the additional logic to be executed before or after the decorated function, and it also calls the decorated function. Finally, the decorator function returns this wrapper.

The key step, as Reuven demonstrated, is then reassigning the original function’s name to the returned wrapper function. For instance, my_function = decorator(my_function). This reassignment effectively replaces the original my_function with the new, enhanced wrapper function, without changing how my_function is called elsewhere in the code. This explicit, step-by-step construction revealed the modularity and power of decorators, highlighting how they can seamlessly inject new behavior into existing functions while preserving their interfaces. Understanding this manual process is fundamental to debugging and truly mastering decorator usage.

Python’s Syntactic Sugar: The @ Operator

Once the manual mechanics of decorators were firmly established, Reuven introduced Python’s elegant and widely adopted @ syntax. This syntactic sugar simplifies the application of decorators significantly, making code more readable and concise. Instead of the explicit reassignment, my_function = decorator(my_function), the @ symbol allows developers to place the decorator name directly above the function definition:

@decorator
def my_function():
#### ...

Lerner emphasized that this @ notation is merely a convenience for the manual wrapping process discussed earlier. It performs the exact same operation of passing my_function to decorator and reassigning the result back to my_function. This clarity was vital, as many developers initially find the @ syntax magical. Reuven illustrated how this streamlined syntax enhances code readability, especially when multiple decorators are applied to a single function, or when creating custom decorators for specific tasks. The @ operator makes decorators a powerful and expressive tool in the Python developer’s toolkit, promoting a clean separation of concerns and encouraging reusable code patterns.

Practical Applications of Decorators

The tutorial progressed into a series of practical examples, showcasing the diverse utility of decorators in real-world scenarios. Reuven presented various use cases, from simple enhancements to more complex functionalities:

• “Shouter” Decorator: A classic example where a decorator modifies the output of a function, perhaps by converting it to uppercase or adding exclamation marks. This demonstrates how decorators can alter the result returned by a function.
• Timing Function Execution: A highly practical application involves using a decorator to measure the execution time of a function. This is invaluable for performance profiling and identifying bottlenecks in code. The decorator would record the start time, execute the function, record the end time, and then print the duration, all without cluttering the original function’s logic.
• Input and Output Validation: Decorators can be used to enforce constraints on function arguments or to validate the return value. For instance, a decorator could ensure that a function only receives positive integers or that its output adheres to a specific format. This promotes data integrity and reduces errors.
• Logging and Authentication: More advanced applications include decorators for logging function calls, handling authentication checks before a function executes, or implementing caching mechanisms to store and retrieve previously computed results.

Through these varied examples, Reuven underscored that decorators are not just an academic curiosity but a powerful tool for injecting cross-cutting concerns (like logging, timing, validation) into functions in a clean, non-intrusive manner. This approach adheres to the “separation of concerns” principle, making code more modular, readable, and easier to maintain.

Decorators with Arguments and Stacking Decorators

Reuven further expanded the attendees’ understanding by demonstrating how to create decorators that accept arguments. This adds another layer of flexibility, allowing decorators to be configured at the time of their application. To achieve this, an outer function is required that takes the decorator’s arguments and then returns the actual decorator function. This creates a triple-nested function structure, where the outermost function handles arguments, the middle function is the actual decorator that takes the decorated function, and the innermost function is the wrapper.

He also covered the concept of “stacking decorators,” where multiple decorators are applied to a single function. When decorators are stacked, they are executed from the bottom up (closest to the function definition) to the top (furthest from the function definition). Each decorator wraps the function that results from the application of the decorator below it. This allows for the sequential application of various functionalities to a single function, building up complex behaviors from smaller, modular units. Reuven carefully explained the order of execution and how the output of one decorator serves as the input for the next, providing a clear mental model for understanding chained decorator behavior.

Preserving Metadata with functools.wraps

A common side effect of using decorators is the loss of the decorated function’s original metadata, such as its name (__name__), docstring (__doc__), and module (__module__). When a decorator replaces the original function with its wrapper, the metadata of the wrapper function is what becomes visible. This can complicate debugging, introspection, and documentation.

Reuven introduced functools.wraps as the standard solution to this problem. functools.wraps is itself a decorator that can be applied to the wrapper function within your custom decorator. When used, it copies the relevant metadata from the original function to the wrapper function, effectively “wrapping” the metadata along with the code.

from functools import wraps

def my_decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
##### ... decorator logic ...
        return func(*args, **kwargs)
    return wrapper

This simple yet crucial addition ensures that decorated functions retain their original identity and documentation, making them behave more like their undecorated counterparts. Reuven stressed the importance of using functools.wraps in all custom decorators to avoid unexpected behavior and maintain code clarity, a best practice for any Python developer working with decorators.

Extending Decorator Concepts: Classes as Decorators and Decorating Classes

Towards the end of the tutorial, Reuven touched upon more advanced decorator patterns, including the use of classes as decorators and the application of decorators to classes themselves.

• Classes as Decorators: While functions are the most common way to define decorators, classes can also serve as decorators. This is achieved by implementing the __call__ method in the class, making instances of the class callable. The __init__ method typically takes the function to be decorated, and the __call__ method acts as the wrapper, executing the decorated function along with any additional logic. This approach can be useful for decorators that need to maintain complex state or have more intricate setup/teardown procedures.
• Decorating Classes: Decorators can also be applied to classes, similar to how they are applied to functions. When a class is decorated, the decorator receives the class object itself as an argument. The decorator can then modify the class, for example, by adding new methods, altering existing ones, or registering the class in some way. This is often used in frameworks for tasks like dependency injection, ORM mapping, or automatically adding mixins.

Reuven’s discussion of these more advanced scenarios demonstrated the full breadth of decorator applicability, showcasing how this powerful feature can be adapted to various architectural patterns and design needs within Python programming. This segment provided a glimpse into how decorators extend beyond simple function wrapping to influence the structure and behavior of entire classes, offering a flexible mechanism for meta-programming.

Links:

• Reuven M. Lerner on LinkedIn
• Reuven M. Lerner’s Professional Website
• LernerPython Website (Courses)
• Video URL

Hashtags: #Python #Decorators #PyConUS2024 #Programming #SoftwareDevelopment #Functions #Closures #PythonTricks #CodeQuality #ReuvenMLerner #LernerConsulting #LernerPython

Watch the video

Introduction

The team from INSEE presents Onyxia, an open-source, Kubernetes-based platform designed to offer flexible, collaborative, and powerful cloud environments for data scientists.

Rethinking Data Science Infrastructure

Traditional local development faces issues like configuration divergence, data duplication, and limited compute resources. Onyxia solves these by offering isolated namespaces, integrated object storage, and a seamless user interface that abstracts Kubernetes and S3 complexities.

Versatile Deployment

With a few clicks, users can launch preconfigured environments — including Jupyter notebooks, VS Code, Postgres, and MLflow — empowering fast innovation without heavy IT overhead. Organizations can extend Onyxia by adding custom services, ensuring future-proof, evolvable data labs.

Success Stories

Adopted across French universities and research labs, Onyxia enables students and professionals alike to work in secure, scalable, and fully-featured environments without managing infrastructure manually.

Conclusion

Onyxia democratizes access to powerful cloud tools for data scientists, streamlining collaboration and fostering innovation.

David Hewitt, a key contributor to the PyO3 library, delivered a comprehensive session at PyConUS 2024, unraveling the mechanics of integrating Rust with Python. As a Python developer for over a decade and a lead maintainer of PyO3, David provided a detailed exploration of how Rust’s power enhances Python’s ecosystem, focusing on PyO3’s role in bridging the two languages. His talk traced the journey of a Python function call to Rust code, offering insights into performance, security, and concurrency, while remaining accessible to those unfamiliar with Rust.

Why Rust in Python?

David began by outlining the motivations for combining Rust with Python, emphasizing Rust’s reliability, performance, and security. Unlike Python, where exceptions can arise unexpectedly, Rust’s structured error handling via pattern matching ensures predictable behavior, reducing debugging challenges. Performance-wise, Rust’s compiled nature offers significant speedups, as seen in libraries like Pydantic, Polars, and Ruff. David highlighted Rust’s security advantages, noting its memory safety features prevent common vulnerabilities found in C or C++, making it a preferred choice for companies like Microsoft and Google. Additionally, Rust’s concurrency model avoids data races, aligning well with Python’s evolving threading capabilities, such as sub-interpreters and free-threading in Python 3.13.

PyO3: Bridging Python and Rust

Central to David’s talk was PyO3, a Rust library that facilitates seamless integration with Python. PyO3 allows developers to write Rust code that runs within a Python program or vice versa, using procedural macros to generate Python-compatible modules. David explained how tools like Maturin and setup-tools-rust simplify project setup, enabling developers to compile Rust code into native libraries that Python imports like standard modules. He emphasized PyO3’s goal of maintaining a low barrier to entry, with comprehensive documentation and a developer guide to assist Python programmers venturing into Rust, ensuring a smooth transition across languages.

Tracing a Function Call

David took the audience on a technical journey, tracing a Python function call through PyO3 to Rust code. Using a simple word-counting function as an example, he showed how a Rust implementation, marked with PyO3’s @pyfunction attribute, mirrors Python’s structure while offering performance gains of 2–4x. He dissected the Python interpreter’s bytecode, revealing how the CALL instruction invokes PyObject_Vectorcall, which resolves to a Rust function pointer via PyO3’s generated code. This “trampoline” handles critical safety measures, such as preventing Rust panics from crashing the Python interpreter and managing the Global Interpreter Lock (GIL) for safe concurrency. David’s step-by-step breakdown clarified how arguments are passed and converted, ensuring seamless execution.

Future of Rust in Python’s Ecosystem

Concluding, David reflected on Rust’s growing adoption in Python, citing over 350 projects monthly uploading Rust code to PyPI, with downloads exceeding 3 billion annually. He predicted that Rust could rival C/C++ in the Python ecosystem within 2–4 years, driven by its reliability and performance. Addressing concurrency, David discussed how PyO3 could adapt to Python’s sub-interpreters and free-threading, potentially enforcing immutability to simplify multithreaded interactions. His vision for PyO3 is to enhance Python’s strengths without replacing it, fostering a symbiotic relationship that empowers developers to leverage Rust’s precision where needed.

Links:

• PyO3 GitHub repository
• Pydantic website
• Pydantic Logfire platform

Hashtags: #Rust #PyO3 #Python #Performance #Security #PyConUS2024 #DavidHewitt #Pydantic #Polars #Ruff