--- published: true layout: post title: 'My Response On The Department Of Veterans Affairs (VA) RFI For The Lighthouse' image: https://s3.amazonaws.com/kinlane-productions2/veterans-affairs/va-logo.jpg ---

I am working with my partners in the government API space (Skylight, 540, Agile Six) to respond to a request for information (RFI) out of the Department of Veterans Affairs (VA), for what they call the Lighthouse API Management platform. The RFI provides a pretty interesting look into the way the government agency which supports our vets is thinking about how they should be delivering government resource using APIs, but also how they play a role in the wider healthcare ecosystem. My team is meeting today to finalize our response to the RFI, and in preparation I wanted to prepare my thoughts, and in my style of doing things, involves publishing them here on API Evangelist.

You can read the whole RFI, but I’ll provide the heart of it, to help set the table for my response.

Introduction: To accelerate better and more responsive service to the Veteran, the Department of Veterans Affairs (VA) is making a deliberate shift towards becoming an Application Programming Interface (API) driven digital enterprise. A cornerstone of this effort is the setup of a strategic Open API Program that is adopting an outside-in, value-to-business driven approach to create APIs that are managed as products to be consumed by developers within and outside of VA.

Objectives: VA has started the process of establishing an API Management Platform, named Lighthouse. The purpose of Lighthouse is to establish the Next Generation Open Digital Platform for Veterans, accelerating the transformation in core domains of VA, such as Health, Benefits, Burial and Memorials. This platform will be a system for designing, developing, publishing, operating, monitoring, analyzing, iterating and optimizing VA’s API ecosystem. These APIs will allow VA to leverage its investment in various digital assets, support application rationalization, and allow it to decouple outdated systems and replace them with new, commercial, off the shelf, Software as a Service (SaaS) solutions. It will enable creation of new, high value experiences for our Veterans, VA’s provider partners, and allow VA’s employees to provide better service to Veterans.

With some insight into how they will achieve those objectives:

They include some bullets for what they see as the road map for the Lighthouse API management platform:

This is all music to my ears. These are objectives, and a road map that I can get behind. It is an RFI that reflects where all federal agencies should be going, but it also is also extra meaningful for me to see this coming out of the VA. Definitely making it something I want to support however I can. The window for responding is short, so I wanted to be able to stop what I’m doing, and give the RFI a proper response, regardless of who ends up running the platform. I’ve done type of RFI storytelling several times in the past, with the FAFSA API for Department of Education, the Recreational Information Database (RIDB) out of Department of Interior, when Commerce hired a new Chief Data Officer, and just in general support of the White House API strategy. This time is different, only because I have a team of professionals who can help me deliver beyond just the RFI.

Some Background On Me And The VA Before I get to the questions included as part of the RFI, I wanted to give some background on my relationship to the VA, and supporting veterans. First, my biological father was career military, and the father that raise me was a two tour Vietnam veteran, exposing me to the veterans administration and hospitals at an early age. He passed away in the 1990s from cancer he developed as a result of his Agent Orange exposure. All this feeds into my passion for applying APIs in this area of our society. Additionally, I used to work for the Department of Veterans affairs, as a Presidential Innovation Fellow in 2013. I didn’t stay for the entire fellowship, exiting during the government shutdown, but I have continued to work on opening up data sets, supporting VA related API conversations, and trying to keep in tune with anything that is going on within the agency. All contributing to this RFI making me pretty happy.

My Answers To The RFI Questions The VA provides 20 separate questions as part of their RFI. Shining a light on some of the thinking that is occurring within the agency. I’ve bolded their questions, and provide some of my thought inline, sharing my official responses. My team will work to combine everyones feedback, and publish a formal response to the VA’s RFI.

1. Drawing on your experience with API platforms, how do you see it being leveraged further within the healthcare industry in such a manner as described above? What strengths and opportunities exist with such an approach in healthcare?

An API first platform as proposed by the VA reflects what is already occurring in the wider healthcare space. With veteran healthcare spending being such a significant portion of healthcare spending in this country, the presence of an API platform like this would not just benefit the VA, veterans, veteran hospitals, and veteran service organizations (VSO), it would benefit the larger economy. A VA API platform would be the seed for a federated approach to not just consuming valuable government resources, but also deploying APIs that will benefit veterans, the VA, as well as the larger healthcare ecosystem.

Some of the strengths of this type of approach to supporting veterans via an open data and API platform, with a centralized API operations strategy would be:

These platform strengths set the stage for some interesting and beneficial opportunities to emerge from within the platform community, but also the wider healthcare ecosystem that already exists, and is evolving, which sill step up and participate and engage with available VA resources. Here are some examples of how this can evolve based upon existing API ecosystem within and outside the healthcare ecosystem:

The benefits of an API platform such as the one proposed as part of this RFI is that they make the delivery of critical services a team effort. Domains within the VA can do what they do best, and benefit from having a central platform, and team to help them deliver consistent, reliable, scalable API access to their resources, for use across web, mobile, device, spreadsheet, analysis, and other applications. Externally, healthcare vendors, hospitals, practitioners, veterans, and the people that support them can all work together to put valuable API resources to work, helping make sense of data, deliver more modular, meaningful applications that all focus on the veteran.

The fact that this RFI exists shows that the VA is looking to keep pace with where the wider healthcare sector is headed. Four our of five of the leading EHR providers have existing API platforms. Demonstrating where the healthcare sector is headed, and something that reflects the vision established in this RFI. The strength and the opportunity with such an approach to delivering healthcare services for veterans will be fully realized when the VA becomes interoperable, and plug and play with the wider healthcare sector.

2. Describe your experience with different deployment architecture supported by MuleSoft (e.g. SaaS only, On Premise Only, Private cloud, Hybrid – Mix of SaaS and Private Cloud) and in what industry or business process it was used? Please include whether your involvement was as a prime or subcontractor, and whether the work was in the commercial or government sector.

My exposure to the Mulesoft in a production environment has been limited in recent years, especially within the last couple of years. However, during my time at the Veterans Affairs, and working in government, I was actively involved with the company, regularly engaging with their product and sales team in several areas:

Based upon my experience working with the Mulesoft team, and through my regular monitoring of their services, tooling, and engaging with some of their clients I am confident in my ability to tailor a platform strategy that would work with their API gateway, IAM, definitions, discovery, and other solutions. I have been one of the analysts in the API sector who studies Mulesoft, and the other API management providers, and understand the strengths, and weaknesses of each of the leading vendors, as well maintain and understanding of how API management is being commoditized, standardized, and applied across all the players. I’m confident this knowledge will transfer to delivering an effective vision for the VA, involving Mulesoft solutions.

3. Describe any alternative API Management Platforms that are offered as SaaS offerings, On Premise, Private Cloud, or Hybrid. Please detail how these solutions can scale to VA’s needs managing approximately 56,000 transactions per second through connecting to VistA and multiple commercial and open source EHRs (conforming to US Core Profiles of the HL7 FHIR standards), multiple commercial Enterprise Resource Planning (ERP) systems, various home grown systems including Veterans Benefit Management Service (VBMS), Corporate Data Warehouse, and VA Time & Attendance System (VATAS), and commercial real-time analytics software packages, and open source tools enabling rapid web and mobile application development.

Monitoring and understanding API management platforms is something I’ve done since 2010 in a full time capacity. I’ve studied the evolution of the enterprise API gateway from its service oriented architecture days, into the cloud as a commodity with AWS, Google, and Azure, as well as the deployment on-premise, in hybrid scenarios, and even on-device. To support my partners, clients, and the readers of my blog, I work regularly to test drive, and understand leading API management solutions available on the market today.

While I study the proprietary and open source API gateway and management solutions out there, I also work to understand where API providers will be operating these solutions, which means having a deep understanding of how native and installed API management occurs in the cloud, hybrid, on-premise, on-device, and anywhere it needs to be in 2017.

In full disclosure, I’ve worked with almost every API management provider out there in some capacity. Similar to providing industry analysis to Mulesoft, I helped WSO2 define and evolve their open source API management solution. Engaged and partnered with 3Scale (now Red Hat), Apigee (Now Google), Mashery (now Tibco), and others. I’ve also engaged with NREL and their development of API Umbrella, as well as the GSA when it came to implementing in support of api.data.gov. I’m currently taking money from Tyk, 3Scale, Runscope, and Restlet–all servicing the space. It is my job to understand the API management playing field, as well as the nuts and bolts of all the leading providers.

While it is important to be able to dive deep and support specific solutions for specific project when it comes to API management, for a platform to handle the scale and scope of the Lighthouse API management platform it will have to be able to provide support for connecting to a robust set of known, and unknown backend systems. While many organizations we’ve worked with strive for a single gateway or API management solution, in reality many often operating multiple solutions, across many vendors, and multiple cloud or on-premise environments. The key to a robust, scalable platform is the ability to easily define, configure, and repeat regularly across environment, providing a consistent API surface area across implementations, groups, backend, and gateway infrastructure.

4. Describe your company’s specific experience and the strategies that you have employed for ensuring the highest level of availability and responsiveness of the platform (include information about configuration based capabilities such as multi-region deployments, dynamic routing, point of presence, flexible/multi-level caching, flexible scaling, traffic throttling etc.).

Our approach to delivering API infrastructure involves assessing scalability at every level of API management within our control. When it comes to API deployment and management we aren’t always in control over every layer of the stack, but we always work to configure, optimize, and scale whenever we possibly can. Every API will be different, and a core team will enjoy a different amount of control over backends, but we always consider the full architectural stack when ensuring availability and responsiveness across API resources, looking at eight common layers:

API stability isn’t purely a technical game. There are plenty of tools at our disposal for scaling, optimizing, and injecting efficiencies into the API life cycle. However, the most important contributor to reliability is experience, and making sure you measure, analyze, and understand the needs of each API. This is where modern approaches to API management come into play, and understand how API consumers are putting resources to work, and optimizing each API to support this at which every layer possible.

5. The experiences you have and the strategies that you have employed to ensure the highest level of security for the platform. Please address policy / procedure level capabilities, capabilities that ensure security of data in transit (e.g. endpoint as well as payload), proactive threat detection etc.

API security is another dimension of the API sector I’ve been monitoring and studying on a regular basis. I have just finished a comprehensive guide to the world of API security, which will be the first guide of its kind when publish next week. I’ve been monitoring general approaches to securing infrastructure, as well as how that impacts API security. I’ve been taking what I’ve found in my research, as well as work with clients, and aggregated into 15 separate areas to consider as we develop a strategy to deliver the high levels of security.

Our API security experience comes from past projects, working with vendors, and studying the best practices of API leaders. Increasingly API security is a group effort, with a growing number of opportunities to work with security organizations, and major tech platforms who see the majority of threats present today. Increasing the volume of information available to integrate directly into platform security operations.

6. Please describe your experience with all the capabilities that the platform offers and the way you have employed them to leverage existing enterprise digital assets (e.g. other integration service buses, REST APIs, SOAP services, databases, libraries)

I have been working with databases for 30 years. I began working with web services in 2000, and have worked exclusively with web APIs in a full time capacity since 2010. I’ve worked hard to keep myself in tune with the protocols, and design patterns in use across the public and private API sectors. Here are the capabilities I’m tuned into currently.

This reflects the core capabilities present across the API landscape. While SOAP and REST make up much of the landscape, hypermedia, GraphQL, event sourcing, and other approaches are seeing more adoption. I emphasize every platform make REST, or web APIs the central focus of the platform, keeping the bar low for API consumers, leverage web technology to reach the widest audience possible.

7. Please describe your experience and strategies that you have employed at enterprises to create Experience Services from mashup/aggregation/combination of other API’s, services, database calls etc.

I’ve been tracking on API aggregation as a discipline for over five years now, having worked with several groups to aggregate common APIs. It is an under developed layer to the web API sector, but one that has a number of proprietary, as well as open source solutions available.

I’ve personally worked on a number of API aggregation project, spanning a handful of business sectors:

These are all the reasons why I was working on API aggregation, and spending time researching the subject. Here are some of the technology, and approaches I have been using to deliver on API aggregation.

In an API utopia, everyone would use common API definitions, media types, and schema. In the real world, everyone does things their own way. API aggregation, and tools like OpenAPI, APIs.json, JSON Schema, and JSON Path allow us to standardize through machine readable definitions that connect the dots for us. API aggregation is still very much a manual process, or something offered by an existing SaaS provider, without much innovation–the tools are out there, we just need more examples we can emulate, and tooling to support.

8. Please describe your experience and strategies for Lifecycle Management of APIs for increasing productivity and enabling VA to deliver high value APIs rapidly, on-boarding app developers and commercial off the shelf applications.

I feel like this overlaps with question number 14, but is more focused on on-boarding, and client perspective of the API lifecycle. Question 14 feels like lifecycle optimization from an API provider perspective, and this is about efficiencies regarding consumption. Maybe I’m wrong, but I am a big believer in the ability of a central API portal, as well as federated portals, to increase API production, because they are delivering the resources consumers need, in a way that allows them to onboard and consume across many developers and commercial software platforms.

My experiences in fulfilling an API lifecycle from the consumer perspective always begins with a central portal, but one that posses all the required API management building blocks to not just deliver aPIs rapidly, incentivize consumption, and feedback in a way that evolves them throughout their life cycle:

Developer Experience (DX) is something that significantly speeds up the overall API lifecycle. Backend teams can efficiently define, deliver, and evolve their APIs without API consumers on-boarding, integrating, and providing feedback on what works, and what doesn’t work. A central API portal strategy for Liqhthouse API management is key to facilitating movement along the API lifecycle, reducing friction, eliminating road blocks, and reducing the chance an API will never full be realized in production.

Lighthouse API management is a central API platform portal, but could also be a forkable DX experience that can be emulated by other internal groups, adding federated edges to the Lighthouse API management platform. Providing a network of consistent API portals, across many different groups, which share a common on0boarding, authentication, documentation, support, communication, and analysis approach. Shifting the lifecycle beyond just a linear start to finish thing, and making something that works as a network of API nodes that all work together as a single platform.

9. Please describe your experience and strategies for establishing effective governance of the APIs.

I’ve been working with a variety of organizations around the topic of API governance for 2-3 years now, and with some recent advances I’m starting to see more API governance strategies mature, and become something that goes well beyond just API design.

API governance is something i’ve tuned into across a variety of enterprise, higher educational institutions, and government agencies. These are the areas I’m researching currently as part of my work as the API Evangelist.

Sadly, API governance isn’t something I’ve seen consistently applied across the enterprise, at institutions and government agencies. There is no standard for API governance. There are few case studies when it comes to API governance to learn from. Slowly I am seeing larger enterprises share their strategies, and seeing some universities publish papers on the topic. Providing some common building blocks we can organize into a coherent API governance strategy.

10. Please describe your experience and methodologies for DevOps/RunOps processes across deployments. Highlight strategies for policy testing, versioning, debugging etc.

API management is a living entity, and we focus on delivering API operations with flat teams who have access to the entire stack, from backend to understanding application and consumer needs. All aspects of the API life cycle embraces a microservices, continuous evolution pace, with Github playing a central from from define to deprecation.

A DevOps focus is maintained across as many stops along an API life cycle, and reflected in API governance practices. However, it is also recognized that a DevOps will not always be compatible with existing legacy practices, and custom approaches might be necessary to maintain specific backend resources, until their practices can be evolved, and brought in alignment with wider practices.

11. Please describe your experience and strategies employed for analytics related to runtime management, performance monitoring, usage tracking, trend analysis.

Logging and anaysis is a fundamental component of API management, feeding an overall awareness of how API are being consumer, which contributes to the product road map, security, and overall platform reliability. The entire API stack should be analyzed from the backend, to the furthest application endpoints, whenever possible.

API management has matured over the last decade to give us a standard approach to managing API access at consumption time, optimizing usage, limiting bad behavior, and incentivizing healthy behavior. API monitoring and testing practices have evolved this perspective of the health, availability, and how APIs are being used from the client perspective. All of this information gets funneled into the road map, refining API gateway plans, rate limits, and other run time systems to adjust and support desired API usage.

12. Please describe your experience with integrating MuleSoft with Enterprise Identity Management, Directory Services and implementing Role Based access.

I do not have any experience in this area with Mulesoft specifically, but have worked in general with IAM and directory solutions in other platforms, and heavy usage on AWS, for securing the API gateway’s interaction with existing backend system.

13. Please describe your experience with generating and documenting API’s and the type of standards they follow. Describe approaches taken for hosting these documentations and keeping them evergreen.

API documentation is always a default aspect of the API platform, and delivered just like code as part of DevOps and CI/CD workflows using Github. With almost every stop along the API life cycle, all documentation is API definition driven, keeping things interactive, with the following elements always in play:

All platform documentation is a living, breathing, element of the ecosystem. If should be versioned, evolved, and deployed along with other supporting microservice artifacts. Mulesoft has documentation to support this approach, as well as there being a suite of open source solutions we can consider to support a variety of different types of APIs.

14. Please describe your proposed complete process lifecycle for publishing high quality, high value APIs with highest speed to market.

Evolving beyond question number 8, and addressing the API provider side of the coin, I wanted to share a complete (enough) view of the life cycle from the API provider perspective. Addressing the needs of backend API teams, as well as the core API team when it comes to delivering usable, reliable, APIs that are active, and enjoy health release cycles. While not entirely a linear approach, here as many of the stops along our proposed API lifecycle, as applied to individual APIs, but applied consistently across the entire platform.

This life cycle will play out over and over for each API published on the Lighthouse API platform. It will independently be executed by API teams for each API they produce, and replayed with each major and minor release of an API. A well defined, and well traveled API life cycle helps ensure consistency across teams, helping enforce compliance, familiarity, and reusability across APIs, no matter which team is behind the facade.

15. Please describe your experience and approach towards establishing a 24x7 technical support team for the users, developers and other stakeholders of the platform.

Support is an essential building block of any successful API platform, as well as a default aspect of every single APIs individual life cycle. We break API platform support into two separate distinct categories.

API support should begin with self-service support, encouraging self-sufficiency of API consumers, and minimizing the amount of platform resources needed to support operations:

Beyond self-service support all API platforms should have multiple direct support channels available to API consumers:

A combination of self-service and direct support channels allows for a resource starved core API team, as well as individual backend teams to manage many developers and applications, across many API resources in an efficient manner. Ensuring API consumers get the answers they are looking for, while ensuring relevant feedback and comments end up back in the product road map, with each appropriate product team.

16. Please describe your experience in establishing metrics and measures for tracking the overall value and performance of the platform.

This is somewhat overlapped with question #11, but I’d say focuses in on the heart of metrics and analysis at the API management level. Understand performance and reliability through the entire stack is critical to platform reliability, but the API management core is all about developing an awareness of how APIs are being consumed, and the value generated as part of this consumption. While performance definitely impacts value, we focus on API management to help us measure and understand consumption of each resource, and understand and measure the value delivered in consumption.

I’ve been studying how API management establishes the metrics needed, and measures and tracks API consumption, trying to understand value generation using the following elements:

API management is the future of public data, content, and other resource management. Understanding API consumption, measuring, analysing, and reporting upon API consumption is required to being able to evolve healthy API resources forward, and develop viable applications, that are putting API resources to use. API management will continue to become the tax system, the royalty system, and value generation capture mechanism across government agencies and public resources in coming years.

17. Please describe your experience and the approach you would take as the API Program Core Team to deploy an effective strategy that will allow VA to distribute the development of API’s across multiple teams and multiple contractor groups while reducing friction, risk and time to market.

A microservice, CI/CID, DevOps approach to providing and consuming APIs has begun to shift the landscape for how API platforms are supporting the API life cycle across many disparate teams, trusted external partners, and even 3rd party application and system developers. Distilling all elements of the API supply chain to modular, well defined components, helps establish a platform that can be centralized, but encouraging a consist