Expectations
See here.
Study
This is a QSOS study.
We evaluate every solution against the criteria and get a final grade.
The solution with the highest score is retained.
Please, refer the to expectations to see how to evaluate every criteria.
LemonLDAP
Presentation
LemonLDAP :: NG is made up of 3 main bricks:
- Manager: dedicated to administrators, allows you to configure LemonLDAP :: NG and explore sessions in graphical mode
- Portal: used for user authentication, provides SAML, OpenID Connect and CAS identity provider services, displays protected applications
- Handlers / Reverse Proxies: Apache modules used to protect applications
The primary functionalities are:
- SAML, CAS and OpenID Connect identity and service provider services
- authentication based on LDAP, Kerberos, SQL, Twitter and other protocols
- Storage of configuration and sessions in database (MySQL, PostgreSQL, LDAP, MongoDB, Redis …)
- a unique authentication system based on secure cookies
- a dynamic menu of applications
- a password reset module
- an account self-creation module
- a module to manage multiple authentication (MFA, 2FA)
- a notification device
- a session explorer
The project site mentions good references mainly in France: National Gendarmerie, Nantes Métropole, etc. with several thousand users.
Notice that LemonLDAP is not certified by the Open ID Foundation. Therefore, if it was a real candidate in this study, it would not be evaluated at all.
Screenshots
Scores
| Criteria | Score | Comment |
|---|---|---|
| Development history | 2 | Created in 2004. |
| Development team | 1 | 6 regular active developers for the last version. |
| Popularity | 1 | Few stars on GitHub, but LINAGORA has seen several companies using it, in particular for Administration services. |
| Contributors community | 1 | Developers are active around OW2 tools. |
| Bugs activity | 1 | Bugs get answers rather quickly but are not all solved. |
| Roadmap | 2 | Managed in GitLAB with tags and milestones. |
| Project Management | 2 | Managed by several persons. |
| Intellectual Property | 2 | Owned by OW2. |
| Distribution | 2 | Totally open source. |
| Single Sign-On (SSO) | 2 | - |
| Single Log-Out (SLO) | 2 | Even if we know there are bugs, SLO is an official feature. |
| OIDC: support for Auth Code flow with PKCE | 2 (x3) = 6 | Supported thanks to a patch LINAGORA submitted in Fall 2020. |
| Support of JWT | 1 (x3) = 3 | Only supports JWT for ID tokens, not access tokens. |
| Specifics for mobile devices | 2 | Thanks to the Auth Code flow with PKCE. |
| Groups information | 2 | Feasible, although the configuration is complex to setup. |
| Identity brokering | 2 (x2) = 4 | See here. |
| Default identity provider (brokering) | 2 (x2) = 4 | - |
| Native identity management | 1 (x2) = 2 | Feasible if we store users in a database, but no user interface to manager users or groups. |
| Users management | 1 | Feasible if we store users in a database, but management is done independently of LemonLDAP (e.g. direct edit on the DB). |
| Groups management | 0 | Nothing mentioned in the documentation. |
| Multi-factor authentication | 2 (x3) = 6 | See here. |
| Password reset | 2 | Supported. |
| Password policy | 2 | Supported. |
| Theme customization | 2 | Already used by LINAGORA. |
| LDAP synchronization | 2 | LemonLDAP can directly authenticate from a LDAP server. |
| SAML support | 2 | LemonLDAP can delegate authentication to a CAS server. |
| Ease of configuration | 1 | The configuration is spread in different locations and remains complicated. |
| Official Docker images | 1 | No official image, just several ones from the community. Linagora managed its own Docker images for it. |
| Compliance with Docker best practices | 1 (x3) = 3 | LINAGORA created a better one. |
| Scaling | 2 (x3) = 6 | Configuration is local, only sessions are stored in DB. |
| Configuration | 0 | Configuration cannot be edited through web services or in a gitops fashion. |
| HTTPS | 2 | No issue with that. |
| Logging | 2 | Logging works overall. |
| Statistics | 2 | Available in the manager console. |
| Multi-tenancy | 0 (x2) = 0 | It is not designed for multi-tenancy. |
| Database | 2 (x2) = 4 | LemonLDAP can work with MongoDB. |
| Monitoring | 1 | Complicated to setup. |
| Software Dependencies | 2 | A database is enough. |
| K8s | 2 | LINAGORA created a Helm package that works well. |
| Documentation | 1 | Lots of documentation, but not easy to understand. |
| Resource Consumption | 2 | It can work with less than 2 GB of memory and 2 CPUs. But under heavy loads, it needs more. |
KeyCloak
Presentation
https://github.com/keycloak/keycloak
https://github.com/keycloak/keycloak-community/
Keycloak is made up of 1 Java webapp:
Keycloak runs on Wildfly application server. JGroups is the library used by Wildfly to discover the nodes and manage the replication of data.
By default JGroups use the multicast to manage the nodes. If the multicast is not available on your network you have to change for another protocol.
Keycloak use Infinispan natively to handle caching mechanism . See Here
Keycloak comes with its own embedded Java-based relational database called H2.
A shared external database like PostgreSQL, MySQL, Oracle, etc. Keycloak requires an external shared database if you want to run in a cluster
The primary functionalities are:
- User Registration
- Social login
- Single Sign-On/Sign-Off across all applications belonging to the same Realm
- 2-factor authentication
- LDAP integration
- Kerberos broker
- OIDC/SAML support
- Customizable themes
Screenshots
Scores
| Criteria | Score | Comment |
|---|---|---|
| Development history | 2 | First commit on github (Jun 30, 2013) |
| Development team | 2 | Red Hat company and community |
| Popularity | 2 | 8600 stars on GitHub - Lot of people are using Keycloak |
| Contributors community | 2 | See here |
| Bugs activity | 2 | See here |
| Roadmap | 1 | See here |
| Project Management | 1 | Managed by Red Hat |
| Intellectual Property | 1 | Apache 2 Licence for community version |
| Distribution | 2 | Open source |
| Single Sign-On (SSO) | 2 | - |
| Single Log-Out (SLO) | 2 | - |
| OIDC: support for Auth Code flow with PKCE | 2 (x3) = 6 | - |
| Support of JWT | 2 (x3) = 6 | - |
| Specifics for mobile devices | 2 | Thanks to the Auth Code flow with PKCE. |
| Groups information | ? | - |
| Identity brokering | 2 (x2) = 4 | See here |
| Default identity provider (brokering) | ? (x2) = ? | - |
| Native identity management | 2 (x2) = 4 | Feasible if we store users in a database |
| Users management | 2 | Feasible if we store users in a database |
| Groups management | 2 | - |
| Multi-factor authentication | 2 (x3) = 6 | Mainly 2 FA with OTP See here |
| Password reset | 2 | Supported. |
| Password policy | 2 | Supported. |
| Theme customization | 2 | See here |
| LDAP synchronization | 2 | Supported. |
| SAML support | 2 | Supported. |
| Ease of configuration | 2 | - |
| Official Docker images | 2 | See here |
| Compliance with Docker best practices | 2 (x3) = 6 | - |
| Scaling | 2 (x3) = 6 | - |
| Configuration | 2 | - |
| HTTPS | 2 | No issue with that. |
| Logging | 2 | Logging works overall. |
| Statistics | 0 | Not a real module dedicated to it, you can retrieve session usage |
| Multi-tenancy | 2 (x2) = 4 | Configuration per Realm. The current storage layer is complex, especially when deployed to multiple-sites. It has a number of scalability issues like the number of realms and clients. Sessions are only kept in-memory, which can be good for performance, but not so great for scaling when you consider a large portion of sessions are idle and unused most of the time. See Here |
| Database | 1 (x2) = 2 | can work with SQL dbs, NoSQL dbs are not supported |
| Monitoring | 2 | Support Prometheus Keycloak Metrics SPI plugin |
| Software Dependencies | 2 | A database is enough. |
| K8s | 2 | See here |
| Documentation | 2 | See here |
| Resource Consumption | 2 | It can work with less than 2 GB of memory and 2 CPUs. But under heavy loads, it needs more. |
KeyCloak-X
KeyCloak-X looks really promising focusing on scaling to millions of sessions and scaling to large number of realms and clients Since he is in preview, it breaks a study constraint and will thus not be evaluated.
Hydra (ORY)
Hydra looks really promising focusing on OAuth 2.0 and OpenID Connect but he is not a real IDP neither support SAML It breaks a study constraint and will thus not be evaluated.
Kratos (ORY)
Kratos is not certified.
It breaks a study constraint and will thus not be evaluated.
OpenAM
Presentation
https://www.openidentityplatform.org/openam
https://github.com/OpenIdentityPlatform/OpenAM
Screenshots
Scores
| Criteria | Score | Comment |
|---|---|---|
| Development history | 2 | Created in 2008 (OpenSSO). |
| Development team | 0 | 2 regular active developers for the last version. |
| Popularity | 0 | 345 stars on GitHub, the project seems not to be used a lot since ForgeRock stop supporting it. |
| Contributors community | 1 | Gitter and Github issues with few activity |
| Bugs activity | 0 | There are opened issues without activity |
| Roadmap | 0 | There are only tags but no roadmap |
| Project Management | 1 | Seems to be managed by 3a systems |
| Intellectual Property | 2 | CDDL |
| Distribution | 2 | Open source |
| Single Sign-On (SSO) | 2 | - |
| Single Log-Out (SLO) | 2 | - |
| OIDC: support for Auth Code flow with PKCE | 2 (x3) = 6 | - |
| Support of JWT | 2 (x3) = 6 | - |
| Specifics for mobile devices | 2 | Thanks to the Auth Code flow with PKCE. |
| Groups information | ? | - |
| Identity brokering | 2 (x2) = 4 | See here |
| Default identity provider (brokering) | ? (x2) = ? | - |
| Native identity management | 2 (x2) = 4 | Feasible if we store users in a database |
| Users management | 2 | Feasible if we store users in a database |
| Groups management | 2 | - |
| Multi-factor authentication | 2 (x3) = 6 | See here |
| Password reset | 2 | Supported. |
| Password policy | 2 | Supported. |
| Theme customization | 1 | Maybe limited customization |
| LDAP synchronization | 2 | Supported. |
| SAML support | 2 | Supported. |
| Ease of configuration | 2 | - |
| Official Docker images | 2 | See here |
| Compliance with Docker best practices | 1 (x3) = 3 | - |
| Scaling | 2 (x3) = 6 | - |
| Configuration | 2 | - |
| HTTPS | 2 | No issue with that. |
| Logging | 2 | Logging works overall. |
| Statistics | 2 | - |
| Multi-tenancy | 2 (x2) = 4 | Configuration per Realm. |
| Database | 2 (x2) = 4 | can work with Cassandra and SQL |
| Monitoring | 1 | isAlive.jsp healthcheck exists |
| Software Dependencies | 2 | A database is enough. |
| K8s | 2 | See here |
| Documentation | 1 | Documentation is hard to find |
| Resource Consumption | 1 | - |
oxAuth (Gluu Server)
Presentation
https://github.com/GluuFederation/oxAuth
Note: SAML is supported through Shibboleth IDP extra dependency
Screenshots
Scores
| Criteria | Score | Comment |
|---|---|---|
| Development history | 2 | First commit on github (Mar 23, 2014) and company (2009) |
| Development team | 0 | Only 1 active developer since the beginning of 2021 |
| Popularity | 0 | 309 stars on GitHub - Seems not a lot of people are using Gluu |
| Contributors community | 1 | - |
| Bugs activity | 1 | - |
| Roadmap | 1 | See here |
| Project Management | 1 | Managed by Gluu company |
| Intellectual Property | 1 | - |
| Distribution | 2 | Open source |
| Single Sign-On (SSO) | 2 | - |
| Single Log-Out (SLO) | 2 | - |
| OIDC: support for Auth Code flow with PKCE | 2 (x3) = 6 | - |
| Support of JWT | 2 (x3) = 6 | - |
| Specifics for mobile devices | 2 | Thanks to the Auth Code flow with PKCE. |
| Groups information | ? | - |
| Identity brokering | 2 (x2) = 4 | See here |
| Default identity provider (brokering) | ? (x2) = ? | - |
| Native identity management | 2 (x2) = 4 | Feasible if we store users in a database |
| Users management | 2 | Feasible if we store users in a database |
| Groups management | 2 | - |
| Multi-factor authentication | 2 (x3) = 6 | See here |
| Password reset | 2 | Supported via oxTrust (only works if password are in backend) |
| Password policy | 0 | Not really Supported. |
| Theme customization | 2 | - |
| LDAP synchronization | 2 | Supported. |
| SAML support | 2 | Supported. |
| Ease of configuration | 1 | - |
| Official Docker images | 2 | See here |
| Compliance with Docker best practices | 2 (x3) = 6 | See here |
| Scaling | 2 (x3) = 6 | - |
| Configuration | 2 | - |
| HTTPS | 2 | No issue with that. |
| Logging | 1 | Logging split in different modules (oxTrust) |
| Statistics | 2 | - |
| Multi-tenancy | 0 (x2) = 0 | Gluu was not built to support multi-tenancy |
| Database | 0 (x2) = 0 | can work with Couchbase and LDAP |
| Monitoring | 0 | See here |
| Software Dependencies | 0 | Lot of dependencies see here |
| K8s | 2 | See here |
| Documentation | 2 | See here |
| Resource Consumption | 1 | - |
WSO2 Identity Server
Presentation
https://wso2.com/identity-and-access-management/
https://github.com/wso2/product-is
Screenshots
Scores
| Criteria | Score | Comment |
|---|---|---|
| Development history | 2 | First commit on github (Apr 20,2014) and company (2005) |
| Development team | 2 | Mainly WSO2 company and community |
| Popularity | 2 | 442 stars on GitHub - Lot of people behind the WSO products https://wso2.com/team/all/ |
| Contributors community | 1 | Slack https://wso2is.slack.com/ with few activity, there are also webinars and events |
| Bugs activity | 1 | - |
| Roadmap | 1 | See here |
| Project Management | 1 | Managed by WSO2 company |
| Intellectual Property | 1 | Apache 2 Licence for community version |
| Distribution | 2 | Open source |
| Single Sign-On (SSO) | 2 | - |
| Single Log-Out (SLO) | 2 | - |
| OIDC: support for Auth Code flow with PKCE | 2 (x3) = 6 | - |
| Support of JWT | 2 (x3) = 6 | - |
| Specifics for mobile devices | 2 | Thanks to the Auth Code flow with PKCE. |
| Groups information | ? | - |
| Identity brokering | 2 (x2) = 4 | See here |
| Default identity provider (brokering) | ? (x2) = ? | - |
| Native identity management | 2 (x2) = 4 | Feasible if we store users in a database |
| Users management | 2 | Feasible if we store users in a database |
| Groups management | 0 | Support only roles |
| Multi-factor authentication | 2 (x3) = 6 | See here |
| Password reset | 2 | Supported. |
| Password policy | 2 | Supported. |
| Theme customization | 1 | Maybe limited customization |
| LDAP synchronization | 2 | Supported. |
| SAML support | 2 | Supported. |
| Ease of configuration | 1 | The configuration is spread in different locations but they work on a new console in last version |
| Official Docker images | 2 | See here |
| Compliance with Docker best practices | 2 (x3) = 6 | - |
| Scaling | 2 (x3) = 6 | - |
| Configuration | 2 | - |
| HTTPS | 2 | No issue with that. |
| Logging | 2 | Logging works overall. |
| Statistics | 2 | - |
| Multi-tenancy | 2 (x2) = 4 | Configuration per Tenant. |
| Database | 1 (x2) = 2 | can work with SQL dbs |
| Monitoring | 2 | Support Prometheus via JMX |
| Software Dependencies | 2 | A database is enough. |
| K8s | 2 | See here |
| Documentation | 2 | See here |
| Resource Consumption | 1 | - |
Synthesis
Open Source Criteria
About the criteria related to Open Source…
| Solutions | Scores |
|---|---|
| LemonLDAP | 14 |
| KeyCloak | 15 |
| Hydra (ORY) | Constraint-breaker |
| Kratos (ORY) | Constraint-breaker |
| WSO2 Identity Server | 13 |
| oxAuth (Gluu) | 9 |
| OpenAM | 8 |
Functional Requirements
About the criteria related to functional requirements…
| Solutions | Scores |
|---|---|
| LemonLDAP | 40 |
| KeyCloak | 42 |
| Hydra (ORY) | Constraint-breaker |
| Kratos (ORY) | Constraint-breaker |
| WSO2 Identity Server | 39 |
| oxAuth (Gluu) | 40 |
| OpenAM | 41 |
Technical Requirements
About the criteria related to technical requirements…
| Solutions | Scores |
|---|---|
| LemonLDAP | 4 |
| KeyCloak | 4 |
| Hydra (ORY) | Constraint-breaker |
| Kratos (ORY) | Constraint-breaker |
| WSO2 Identity Server | 4 |
| oxAuth (Gluu) | 4 |
| OpenAM | 4 |
Requirements for Developers
About the criteria related to requirements for developers…
| Solutions | Scores |
|---|---|
| LemonLDAP | 1 |
| KeyCloak | 2 |
| Hydra (ORY) | Constraint-breaker |
| Kratos (ORY) | Constraint-breaker |
| WSO2 Identity Server | 1 |
| oxAuth (Gluu) | 1 |
| OpenAM | 2 |
Requirements for Operations
About the criteria related to requirements for operations…
| Solutions | Scores |
|---|---|
| LemonLDAP | 28 |
| KeyCloak | 36 |
| Hydra (ORY) | Constraint-breaker |
| Kratos (ORY) | Constraint-breaker |
| WSO2 Identity Server | 37 |
| oxAuth (Gluu) | 26 |
| OpenAM | 34 |
Global Scores
| Solutions | Scores |
|---|---|
| LemonLDAP | 87 |
| KeyCloak | 99 |
| Hydra (ORY) | Constraint-breaker |
| Kratos (ORY) | Constraint-breaker |
| WSO2 Identity Server | 94 |
| oxAuth (Gluu) | 80 |
| OpenAM | 89 |
Conclusion
Given the context, KeyCloak gets the best score.
It does not mean it is perfect: it is not the best solution for multi-tenancy (no native design for that, the internal model must be tweaked)
and it will meet scalability issues. It can handle thousands of connections but not millions.
Anyway, it is suitable and will already solve issues that LINAGORA had with LemonLDAP.
The way it will be deployed (K8s or VM) is not decided yet.