My Rust Development Journey
Introduction
In July of 2022, I decided to learn Rust after having been a Java developer throughout my entire career. I wanted to step out of the Java landscape and try something new. Within the JVM landscape, I’ve been using Groovy for writing tests for years. I also went through a Scala phase, which introduced me to functional programming (technically, the first FP I did was in XSLT, but let’s not count that here). More recently, I’ve been using Kotlin at my day job. It’s a decent language, but I largely fail to see the point.
There has been quite a buzz around Rust for some time, and it’s completely outside the sphere of languages I’m comfortable with. So, I thought I’d give it a try. I took the Rustlings course, which provided a good introduction to the language. Following that, I thought it would be a good exercise to develop something significant to explore what I could achieve with the language. I have a personal Java “playground” application for experimenting with technologies and programming techniques, and I thought it would be worthwhile to port this to Rust. In doing so, I could attempt to recreate the best practices and patterns for building services in another language. Since beginning this endeavour, I have written, edited, or deleted over 40,000 lines of Rust code, and my codebase currently contains around 12,000 lines of code.
I understand that Rust is not Java, and Java is not Rust. I’ve learned that what you do in one language is not necessarily the way to do it in another language. I’ve also learned where and how I can use object-oriented programming in Rust, and where it just isn’t necessary. Unlearning the object-first approach of Java has been a challenge, but it has been worth it. This document makes an effort to detail what I wanted to achieve and how I approached it.
Domain Driven Design
Over the last few years, my development focus has been on Domain-Driven Design (DDD) practices. I had experimented with some of the techniques without realising it while using the Lagom Framework and learning about Command Query Responsibility Segregation (CQRS) and Event Sourcing (ES). As a learning exercise, I even wrote my own CQRS/ES framework using Vert.x. Although I don’t currently require event sourcing, I strongly believe that command-query segregation is vital to building systems that are easy to understand and maintain.
It was a top priority for me to be able to incorporate DDD concepts into Rust. In a previous post, I discussed the implementation of the specification pattern, but I also needed to explore other parts.
Value objects, identities, entities and aggregates
When I first started learning about Domain-Driven Design (DDD), I discovered the Cargo example, which I found easy to follow. I appreciated the class hierarchy, which included an interface for value objects, an interface for entities, and a composite specification pattern. Using this as a basis, I expanded on it by:
- Defining an
Identityinterface that extendedValueObject - Having
Entitytake anIdentityas a generic parameter - Replacing
AbstractSpecificationwith default methods in theSpecificationinterface
Additionally, I came across a CQRS/ES framework written in C# that had an Aggregate object similar to an Entity, but allowed for a list of events to be “flushed” and consumed. Given my preference for immutable objects, I also borrowed this idea.
Repositories
An entity repository should only contain CRUD operations, and not even list operations. I prefer to keep my read model operations separate. It is important to ensure that the repository definition is generic and deals with futures.
I was enlightened by an article on Enterprise Craftsmanship that extolled the virtues of leaving uniqueness checks to the database and handling errors appropriately. I see the point here: why write code to ensure that the username you have input is unique for create or update when the database can inform you of a violation when you try to write it? I had already achieved this in Java, so it was important for me to be able to replicate the behaviour in Rust using the library of my choice.
Commands, events and queries
My personal development work is done in Vert.x. I have developed a DDD library despatching commands, events and queries via the Vert.x event bus. I’m looking for something similar in the Rust world. I like how using an event bus allows you to completely decouple your code.
Tool Belt
With the requirements in place I set about building a tool belt. This would provide the building blocks upon which I would write services in Rust. I learned a lot about the language in the process. I made liberal use of the linter. I still do. What surprised me the most was, while I learned what practices from Java I could apply to Rust (and ones that were unnecessary), I also found what I was learning informed my Java code too. There were many instances where I had over-complicated my Java code (several layers of abstract class being a good example) and was able to rewrite a huge amount to make it simpler - or at least more understandable. I’ve been very interested in writing leaner code for a long time and, weirdly, writing in Rust gave me permission to do that in Java.
What gave me the biggest sense of accomplishment was being able to implement what I thought would be some of the more “tricky” features in Rust. Interpreting Postgres constraint failures into appropriate errors at the database layer was the first one most important one. Dynamically building SQL queries using the specification pattern was another. This one in particular gave me a few headaches around lifecycles of borrowed objects until I learned to let go of shoe-horning object-oriented Java practices into Rust where it wasn’t necessary and just embrace procedural programming.
Consumer-friendly logging
One thing that I appreciate about the code base at my current place of employment is their consumer-friendly logging approach. This approach ensures that anything logged can be easily consumed by tools like SumoLogic, making searching later more efficient. Additionally, using Logger in your code is prohibited, which prevents loggers from being littered all over the place:
private static final Logger LOGGER = LoggerFactory.getLogger(MyClass.class);Instead, we have a Monitor class that allows you to build an error in a fluent way, like this:
Monitor.alert()
.unexpected()
.process("Get Widget Data")
.param("widgetId", widgetId)
.param("reason", e.getMessage())
.build();This would generate a log message along these lines:
ERROR process=Get Widget Data param=widgetId reason=Widget database is offlineI implemented something similar in Rust. I also added some default data into the log message for debugging purposes. So the following code:
Monitor::trace()
.process("test")
.message("This is a test")
.param("param1", "value")
.log();produces this logged output:
2023-08-20T10:52:33.352959Z TRACE rmg_monitor: target="service.trace" environment=local hostname=Macbook message=This is a test param1=value process_id=test service=service thread=tests::logging_test timestamp=2023-08-20T10:52:33.352824+00:00DDD Library
While it is true that a library is not strictly necessary to provide DDD functionality, I prefer to follow certain patterns and enforce consistency. Upon researching available DDD libraries, I found that 4dk was the only one that came close to providing the functionality I wanted. However, I encountered an issue with this library’s non-asynchronous handlers, as the Sqlx example showed that the repository implementation blocks to get the result. To maintain asynchronicity, I used this as a starting point for my own asynchronous DDD library.
My intention is to try using Tokio’s message channels as a substitute for an event bus, although finding the best approach for this has proven difficult at the moment. My reason for doing this is to decouple the dispatching of messages generated by a message handler from the process that calls it. This will allow a long-running choreographed process to run without the caller waiting for the result. While it may seem simple in theory to just spawn a task, in practice a single initiating event could trigger many commands, each triggering further events and commands. I believe it will take some study, time, and experimentation to get this right.
Uniqueness Error Handling
I was able to meet the requirement described above regarding leaving uniqueness checks to the database by implementing a method that converts a sqlx::Error to an eyre::Report that is returned by the calling repository method. I defined all of my database constraints in an enumeration and matched the constraint name in the error message returned from Sqlx. Then, I used this information to construct a corresponding error, as demonstrated in the following code:
pub fn transform_error(error: Error) -> Report {
match error {
Error::Database(e) => match Constraint::iterator()
.find(|constraint| e.message().contains(constraint.key().as_str()))
{
None => Report::new(e),
Some(constraint) => {
let e: Box<PgDatabaseError> = e.downcast();
constraint.to_report(&e)
}
},
_ => Report::new(error),
}
}
enum Constraint {
PageIdUnique,
PageNotFound,
PageSlugUnique,
}
impl Constraint {
fn iterator() -> Iter<'static, Self> {
static ITEMS: [Constraint; 3] = [
Constraint::PageIdUnique,
Constraint::PageNotFound,
Constraint::PageSlugUnique,
];
ITEMS.iter()
}
pub fn key(&self) -> String {
match self {
Self::PageIdUnique => String::from("cms_page_id_unique"),
Self::PageNotFound => String::from("cms_page_not_found"),
Self::PageSlugUnique => String::from("cms_page_slug_unique"),
}
}
fn to_report(&self, e: &PgDatabaseError) -> Report {
match self {
Self::PageIdUnique => Report::new(PageError::PageExists {
page_id: PageId::from(extract_uuid(e)),
}),
Self::PageNotFound => Report::new(PageError::PageNotFound {
page_id: PageId::from(extract_uuid(e)),
}),
Self::PageSlugUnique => Report::new(PageError::PageSlugInUse {
slug: extract_value(e),
}),
}
}
}
fn extract_uuid(e: &PgDatabaseError) -> Uuid {
e.detail().map_or_else(|| {
alert("extract_uuid", "No detail in error");
Uuid::nil()
}, |val| match Regex::new(
r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}",
) {
Ok(re) => re.find(val).map_or_else(|| {
alert("extract_uuid", "No uuid in error");
Uuid::nil()
}, |m| match Uuid::from_str(m.as_str()) {
Ok(uuid) => uuid,
Err(e) => {
alert("extract_uuid", &e.to_string());
Uuid::nil()
}
}),
Err(error) => {
alert("extract_uuid", &error.to_string());
Uuid::nil()
}
})
}
fn extract_value(e: &PgDatabaseError) -> String {
e.detail().map_or_else(|| {
alert("extract_value", "No detail in error");
String::new()
}, |val| match Regex::new(r"=\(.*\)") {
Ok(re) => re.find(val).map_or_else(|| {
alert("extract_value", &e.to_string());
String::new()
}, |mat| {
let value = mat.as_str();
value[2..value.len() - 1].to_string()
}),
Err(error) => {
alert("extract_value", &error.to_string());
String::new()
}
})
}
fn alert(process: &str, message: &str) {
Monitor::unexpected()
.process(format!("infrastructure::shared::error::{process}").as_str())
.message(message)
.log();
}Web Layer
There are many web frameworks available for Rust, including Poem, Rocket, and Warp, all of which are popular. However, I ended up choosing Actix, although I can’t quite remember why. I think I came across it while reading about the Actix actor framework, and I had also seen it mentioned in a few other library integration examples.
Building the web required the biggest mindset shift for me, coming from a JVM background. Although I had a relatively decent grasp of when to apply object-oriented principles in Rust and when not to, web development is purely procedural. Fortunately, I was able to reference some decent examples in order to accomplish everything I needed to do.
Health Checks
The first features I built in Actix were a ping endpoint for checking if the service is running, and support for health checks. Implementing the ping endpoint was relatively simple. Here’s how I did it:
pub async fn ping(_req: HttpRequest) -> &'static str {
Monitor::support()
.process("actuator::route::ping")
.log();
"PONG"
}I wanted to implement a health check mechanism similar to what is available in Vert.x or Spring. I began by defining my requirements, and my implementation is heavily influenced by Vert.x.
use std::collections::BTreeMap;
use futures::future::LocalBoxFuture;
use serde::Serialize;
#[derive(Serialize)]
pub struct CheckResult {
pub id: String,
pub status: Status,
#[serde(skip_serializing_if = "Option::is_none")]
pub checks: Option<Vec<CheckResult>>,
}
unsafe impl Send for CheckResult {}
#[derive(Serialize)]
pub struct Status {
pub ok: bool,
#[serde(skip_serializing_if = "Option::is_none")]
pub data: Option<BTreeMap<String, String>>,
}
impl Status {
pub const fn ok() -> Self {
Self {
ok: true,
data: None,
}
}
pub fn ok_data(data: BTreeMap<String, String>) -> Self {
Self {
ok: true,
data: Some(data),
}
}
pub const fn ko() -> Self {
Self {
ok: false,
data: None,
}
}
pub fn ko_data(data: BTreeMap<String, String>) -> Self {
Self {
ok: false,
data: Some(data),
}
}
}
pub trait HealthCheck: Send {
fn id(&self) -> String;
fn check(&self) -> LocalBoxFuture<CheckResult>;
}As an example of implementing a health check, here is my implementation for Postgres:
#[derive(new)]
pub struct PostgresHealthCheck {
pool: Rc<PgPool>,
}
unsafe impl Send for PostgresHealthCheck {}
impl HealthCheck for PostgresHealthCheck {
fn id(&self) -> String {
String::from("postgres")
}
fn check(&self) -> LocalBoxFuture<CheckResult> {
Box::pin(
sqlx::query("select 1")
.fetch_one(&*self.pool)
.then(move |result| match result {
Ok(_) => Box::pin(future::ready(CheckResult {
id: self.id(),
status: Status::ok(),
checks: None,
})),
Err(err) => {
let mut data = BTreeMap::<String, String>::new();
data.insert(String::from("error"), Report::new(err).to_string());
Box::pin(future::ready(CheckResult {
id: self.id(),
status: Status::ko_data(data),
checks: None,
}))
}
}),
)
}
}Health checks are executed and consolidated using this method:
pub async fn get_health_status(data: Data<Context>) -> HttpResponse {
Monitor::support()
.process("actuator::route::get_health_status")
.log();
let checks: Vec<Box<dyn HealthCheck>> =
vec![Box::new(PostgresHealthCheck::new(data.pool.clone()))];
let results = join_all(checks.iter().map(|check| check.check())).await;
let result = CheckResult {
id: String::from(crate_name!()),
status: if results.iter().any(|result| !result.status.ok) {
Status::ko()
} else {
Status::ok()
},
checks: Some(results),
};
if result.status.ok {
HttpResponse::Ok()
.insert_header(ContentType::json())
.body(serde_json::json!(result).to_string())
} else {
HttpResponse::ServiceUnavailable()
.insert_header(ContentType::json())
.body(serde_json::json!(result).to_string())
}
}Finally, the health check endpoints are configured as follows:
pub fn configure_health_routes(cfg: &mut web::ServiceConfig) {
cfg.service(web::resource("/ping").route(web::get().to(ping)));
cfg.service(web::resource("/health").route(web::get().to(get_health_status)));
}OpenID Connect
I required support for authentication and authorisation in my reference application to enable fine-grained access control through OpenID Connect, with service-level roles. Keycloak was my choice for identity and access management and I needed to integrate it into my service. Despite my initial reluctance, I ended up developing my own implementation using the actix-web-grants, actix-web-httpauth, and openid libraries. I was unable to find a suitable library that met my requirements and found it simpler to write my own integration rather than try to adapt to an existing solution.
The claims in the JWT token, which has fine-grained access control generated by Keycloak, follow this structure:
{
"aud": "my-service",
"sub": "c215f378-8085-43c8-b627-27f949ef7a76",
"resource_access": {
"my-service": {
"roles": [
"edit_pages",
"read_pages",
"add_pages",
"browse_pages"
]
}
},
"scope": "email profile openid my-service",
"iss": "http://localhost:9999/auth/realms/test",
"name": ...,
"preferred_username": ...,
"exp": 1666586587,
"given_name": ...,
"iat": 1666572187,
"family_name": ...,
"email": ...
}I had to implement a custom Claims trait to support roles in the JWT token. Here’s what I came up with:
#[derive(Deserialize, Serialize, Debug, Clone, Eq, PartialEq)]
pub struct MyClaims {
#[serde(flatten)]
pub standard_claims: StandardClaims,
pub resource_access: BTreeMap<String, BTreeMap<String, Vec<String>>>,
}
impl Claims for MyClaims {
fn iss(&self) -> &Url {
self.standard_claims.iss()
}
fn sub(&self) -> &str {
self.standard_claims.sub()
}
fn aud(&self) -> &SingleOrMultiple<String> {
self.standard_claims.aud()
}
fn exp(&self) -> i64 {
self.standard_claims.exp()
}
fn iat(&self) -> i64 {
self.standard_claims.iat()
}
fn auth_time(&self) -> Option<i64> {
self.standard_claims.auth_time()
}
fn nonce(&self) -> Option<&String> {
self.standard_claims.nonce()
}
fn at_hash(&self) -> Option<&String> {
self.standard_claims.at_hash()
}
fn c_hash(&self) -> Option<&String> {
self.standard_claims.c_hash()
}
fn acr(&self) -> Option<&String> {
self.standard_claims.acr()
}
fn amr(&self) -> Option<&Vec<String>> {
self.standard_claims.amr()
}
fn azp(&self) -> Option<&String> {
self.standard_claims.azp()
}
fn userinfo(&self) -> &Userinfo {
self.standard_claims.userinfo()
}
}
impl CompactJson for MyClaims {}I had to write two methods for authenticating and authorising the JWT token. The first method is used as the parameter for constructing a actix_web_httpauth::middleware::HttpAuthentication instance, which ensures that the JWT token is valid:
pub async fn validator(
req: ServiceRequest,
credentials: BearerAuth,
) -> Result<ServiceRequest, (actix_web::Error, ServiceRequest)> {
let client = req
.app_data::<web::Data<Context>>()
.map(|context| context.oauth_client.clone());
match client {
None => Err((actix_web::Error::from(build_problem(
StatusCode::UNAUTHORIZED,
"OpenID not configured",
)), req)),
Some(client) => {
match authorise(client, credentials.token().to_string()) {
Ok(id_token) => {
if let IdToken::<MyClaims>::Decoded { header: _, payload } = id_token {
req.extensions_mut().insert(payload);
}
Ok(req)
},
Err(error) => Err((actix_web::Error::from(error), req))
}
}
}
}
fn authorise(
client: Rc<Client<Discovered, MyClaims>>,
token: String
) -> Result<IdToken<MyClaims>, Problem> {
let bearer = Bearer {
access_token: String::new(),
scope: None,
refresh_token: None,
expires: None,
id_token: Some(token),
};
let mut token = Token::<MyClaims>::from(bearer);
match token.id_token.as_mut() {
None => Err(build_problem(
StatusCode::UNAUTHORIZED,
"id_token not provided",
)),
Some(id_token) => {
client.decode_token(id_token).map_err(|error| build_problem(
StatusCode::UNAUTHORIZED,
&format!("{error}"),
))?;
client.validate_token(id_token, None, None).map_err(|error| build_problem(
StatusCode::UNAUTHORIZED,
&format!("{error}"),
))?;
Ok(id_token.clone())
}
}
}You may observe that the valid, decoded JWT token is inserted into the HTTP request’s extensions. This is to avoid a second decoding of the JWT token during the authorisation stage, which retrieves the roles from the token. This authorisation method is provided as a parameter to create an instance of actix_web_grants::GrantsMiddleware.
pub async fn extract_claims(req: &ServiceRequest) -> Result<Vec<String>, actix_web::Error> {
let mut extentions = req.extensions_mut();
let claims: Option<MyClaims> = extentions.remove();
match claims {
None => Ok(vec![]),
Some(claims) => {
extentions.insert(claims.clone());
claims.resource_access.get("my-service")
.map_or_else(|| Ok(vec![]), |scope| scope.get("roles")
.map_or_else(|| Ok(vec![]), |roles| Ok(roles.clone())),
)
}
}
}In this sample, the client ID (also the aud claim) is hard-coded as my-service to match the JWT token example shown above. However, in reality, the client ID should match the service name and be obtained from the application configuration. My code is written in a more general manner so that it can be reused across multiple services.
When it comes to configuring a route with authentication and authorisation, I just had to apply the wrap function with the implemented middleware in the correct order:
actix_web::App::new()
...
.service(web::scope("/pages")
.wrap(GrantsMiddleware::with_extractor(extract_claims))
.wrap(HttpAuthentication::bearer(validator))
.configure(configure_page_routes))To restrict an endpoint to users with a specific role, a permission guard is added to the route. Here is an example of how the /pages route is configured for GET and POST:
pub fn configure_page_routes(cfg: &mut web::ServiceConfig) {
cfg.service(web::resource("")
.route(web::post().to(create_page)
.guard(PermissionGuard::new("add_pages")))
.route(web::post().to(forbidden))
.route(web::get().to(list_pages)
.guard(PermissionGuard::new("browse_pages")))
.route(web::get().to(forbidden))
);
}Tying it all together
Command Line Parsing and Service Configuration
I chose clap for command line parsing because it was the first one I came across and it does the job well. It also generates a nice help output. I used macros to extract the service name and version from Cargo.toml.
For configuring services, I used the config crate. There are several ways a service can obtain its configuration, so providing options is useful:
- By default, the configuration file is located in the same directory as the executable. To change this, use the
-bcommand line switch. - The first required configuration is
Application.toml. - For environment-specific configuration files, use
Application-{env}.toml. To use an environment-specific configuration file, run the application with the command line switch-e {env}. - To use a configuration file in a completely different location, use the
-ccommand line switch.
I have not yet added support for environmental variables, but I plan to do so when time permits. This would be the highest level of configuration and would override the other levels of configuration.
The help output looks like this:
❯ ./target/debug/my-service -h
My Rust-powered service.
Usage: my-service [OPTIONS]
Options:
-b, --config-base <config-base>
Path to base configuration directory
-c, --config-location <config-location>
Path to custom configuration file
-e, --env <env>
Environment in which the service is running (development, production, etc. [default: local]
-h, --help
Print help
-V, --version
Print versionApplication Context
I first encountered the concept of an application context when I began using Spring, but it is not limited to the Spring ecosystem. Even after I decided to stop using Spring, I continued to use the concept for configuring a service, creating classes, and wiring them together. The application context also fits with Actix root level data. For my sandbox service, the application context looks like this:
pub struct Context {
pub config: Config, // service configuration
pub pool: Rc<PgPool>, // the SQLx database pool, exposed in the context for service health check
pub oauth_client: Rc<Client<Discovered, MyClaims>>, // OAuth2 client
pub bus: Bus, // bus controlling application services
}This is built in the main method for the service. The extent of this method looks like this:
#[actix_web::main]
async fn main() -> std::io::Result<()> {
setup_tracing()?;
let matches = arg_matches();
let config = match configuration(&matches) {
Ok(config) => config,
Err(error) => return Err(Error::new(ErrorKind::Other, error.to_string())),
};
let oauth2_client = match oauth2_client(&config).await {
Ok(oauth2_client) => oauth2_client,
Err(error) => return Err(Error::new(ErrorKind::Other, error.to_string())),
};
let pool = match pool(&config).await {
Ok(pool) => pool,
Err(error) => return Err(Error::new(ErrorKind::Other, error.to_string())),
};
let host = host(&config);
let port = port(&config);
HttpServer::new(move || {
App::new()
.wrap(TracingLogger::default())
.app_data(web::Data::new(Context::new(
config.clone(),
Rc::new(pool.clone()),
Rc::new(oauth2_client.clone()),
)))
.service(web::scope("/actuator").configure(configure_health_routes))
.service(web::scope("/pages")
.wrap(GrantsMiddleware::with_extractor(extract_claims))
.wrap(HttpAuthentication::bearer(validator))
.configure(configure_page_routes))
})
.bind((host, port))?
.run()
.await
}The Bus object is created within the Context::new method, which is not shown here. Furthermore, this document does not illustrate the methods used to set up various supporting structures for the service, such as the configuration and database pool. These methods are pretty standard.
Tracing
You may have noticed the TracerLogger configuration in the code. I have used OpenTelemetry to instrument I/O bound operations and HTTP requests. This is now a requirement for services, so ensuring tracing is supported is crucial. My viewer of choice, at least for local testing is Jaeger.
Next Steps
I am satisfied with my progress in Rust and in establishing some patterns for service development, as well as deciding where to go from here. As I previously mentioned, I would like to improve the bus to make it more like a traditional bus. However, there are also other additions that I would like to make.
OpenAPI Documentation
I would like to add support for automatically generating OpenAPI specifications for services. To do this, I plan to investigate Utoipa.
WebAssembly
I usually build UIs using TypeScript. For tinkering, my preferred framework is Vue, which I discovered a couple of years ago and really enjoyed. Although I’m not a front-end developer by profession, I’ve observed the fragmentation and fast pace of change in the field over the years.
I am interested in experimenting with WebAssembly for front-end development because I like the concept of using the same language for full-stack development.
In summary, since I started developing in Rust, I have developed a collection of patterns for building services that can handle large scale operations. I have gained an understanding of how Rust communicates with databases, performs asynchronous operations, and when to avoid object-oriented thinking that come with years of Java development. I have experienced the frustration of working with the borrow checker, but have also experienced the elation of finding solutions to these challenges. Furthermore, I have come to appreciate the differences between Rust and the JVM ecosystem, and find Rust enumerations to be particularly wonderful to work with. In fact, I sincerely wish Java had an equivalent to Rust enumerations.
Banner image by Rémi Jacquaint on Unsplash