Causes of the diversity imbalance

March 18, 2016March 22, 2016 Stephen RoughleyDiversity, Equality, Feminism, Racism1 Comment

Throughout the West there are certain groups of people that are underrepresented within the software industry and who are at a disadvantage when it comes to competing for jobs in software development. This is a fact known by many software professionals and there is some great work going on to help correct the imbalance. Despite this, there is still resistance to this work and there is even resistance to accept or even acknowledge the underlying issues that cause the imbalance. Experience has shown me that much of this resistance is down to a lack of easily accessible information and knowledge on the subject matter. Even those I have spoken to that don’t resist the ideas by and large seem to be less informed on the subject than I would have hoped and I have struggled to find comprehensive and accessible sources of information that cover the wide ranging issues at play. Much of the literature on the diversity imbalance and its underlying causes is very complex, certainly too much to cover in depth here and I see this as being a large part of the problem. With that in mind I’d like to take a moment of your time to share a high-level look at some of the underlying issues that drive the underrepresentation of disadvantaged groups.

Just as a heads up, this is written by a software developer living in Britain and so I will be using mostly examples from Britain and there will of course be a focus on software development and the STEM fields but most of the issues here are globally applicable. We are humans wherever we live and work, for good and bad. Furthermore, while I have worked to make this a comprehensive article from the outset, it is by no means intended to be a finished doctoral submission, I want this to be simpler and accessible. I am still researching this material myself in my own time and so I would ask you to treat this as the start of a living document that will continue to be updated as I learn more and uncover more information to share here. If you have anything to add then I would be grateful to hear from you!

A knowledge gap

There is plenty going on in the software development world to bring to attention the issues surrounding the lack of diversity. I’ll give a few examples:

The prominent Martin Fowler covers an excellent introduction to the issues faced by Historically Discriminated Against groups in software development: http://martinfowler.com/bliki/HistoricallyDiscriminatedAgainst.html and discusses the merits of diversity beyond its inherent ethical goodness: http://martinfowler.com/bliki/DiversityImbalance.html
The U.S. tech industry continues to become more motivated to improve the diversity imbalance: http://www.forbes.com/sites/bonniemarcus/2015/08/12/the-lack-of-diversity-in-tech-is-a-cultural-issue
Elise Wei recently discussed how to improve diversity within an organisation: https://medium.com/@EliseWei/diversity-and-the-tech-worker-life-cycle-6605e504854f#.xrdcoulwl
There is a lot of good stuff going on in the developer community such as Black Girls Code: http://techcrunch.com/2016/02/01/black-girls-code-founder-kimberly-bryant-on-racism-and-implicit-bias/ and Code First: Girls in the UK: http://www.codefirstgirls.org.uk/

Despite all of this, I still see a gap here where a lack of knowledge and even a lack of exposure in general to the driving issues behind this movement is creating an obstacle.

Resistance

I’ll start with a point that really must be driven home: there is no evidence that people from one group should be any more likely than those from another group to be attracted to and to naturally succeed in any given field (i.e. STEM). It feels weird that I should be compelled to explain what should be a widely understood truth but it is something that comes up time and time again. This belief, this misconception, is very dangerous as has been proven throughout history and by humanity’s great atrocities. This misconception has been proven to be unsubstantiated by research in the human genome project:

http://amjca.blogspot.co.uk/2009/04/human-genome-project-confirms-that-race.html

Furthermore, modern research is turning up more and more evidence that male and female brains aren’t actually all that different and a lot of the differences we do see result from social pressures:

http://www.sciencemag.org/news/2015/11/brains-men-and-women-aren-t-really-different-study-finds

This in itself should be enough for a reasonable person to realise that there are forces at work beyond ‘natural selection’ which are driving the diversity imbalance. But this is not always the case. Martin Fowler touches on this failing in his articles, where he expresses dismay at otherwise reasonable people falling for tautological reasoning to explain the lack of women in software development. I understand and share Martin’s frustration but we must be careful not to accuse anyone of any misgivings here as this might only serve to steel any existing resistance.

Having taken an interest in social equality for some time now I have often been surprised at where I find resistance to the ideas behind the work to improve the balance of diversity. Over the years I have heard such resistance from some very intelligent people and so I feel we should treat this as a significant part of the overall issue and something we must also overcome on the road to equality. I don’t want to blame anyone for failing to understand the issues here as they are complex and in my experience most people tend not to be encouraged to think about these kinds of issues on a regular basis. In fact it takes some dedicated consideration and research to really get a grasp of the underlying problems and so I don’t think it is fair to accuse anyone of any shortcomings in their resistance here. Instead we should look at this as a chance to open up further dialog around the issues.

The crux of resistance

Ultimately the resistance I encounter when discussing work to improve diversity seems to boil down to a misunderstanding of its raison d’etre . Why is the lack of diversity a problem that we must actively address? I’d like to start with one point which was made by Martin Fowler in his Thoughtworks article ‘Historically Discriminated Against’:

“It takes many generations to undo the effect of centuries of discrimination, so just because the law and society are beginning to catch up doesn’t mean the work of supporting historically-discriminated-against groups can stop.”

What is this effect that Martin speaks of, why is there still a lack of diversity and why do those from disadvantaged groups require direct help?

I’ll use the term ‘disadvantaged’ in relation to these groups from here on in because the issues faced by ‘historically discriminated against’ groups also affect groups that might not fall under that term coined by Thoughtworks. An example of this is the communities of the white working class ethnic group in Britain that have been politically and socially neglected to the point that they are now seeing families that are third generation unemployed and suffering the full force of the ill effects of poverty. There is clear evidence that this group is facing widespread discrimination but their situation is relatively new and many of the forces and factors involved in the failure of people from this group to succeed are still being learned and so I feel the term ‘disadvantaged’ is most appropriate.

In all cases the issues at play here are very complex and so rather than dig deep into the subject matter, I will summarize and list the key points, providing further reading for you to explore.

The points

So why do those from disadvantaged groups require active assistance? Let’s start with the low hanging fruit. Perhaps still the largest, certainly one of the most debilitating, issue that people from disadvantaged groups can face is the cycle of poverty:

https://en.wikipedia.org/wiki/Cycle_of_poverty

It is a well-known phenomenon and affects those who have suffered centuries of discrimination such as African and Native Americans as well as those who are experiencing a more recent trend of poverty such as the British white working class. To summarise, the phenomenon works like this: when you are born into a community where no one in your family or social group has succeeded then the odds are severely stacked against you succeeding. The impact of poverty is of course much more complex however than a simple limit on the success of those affected. It has considerable implications for the mental and physical health and even life expectancy of those trapped by it:

http://www.ascd.org/publications/books/109074/chapters/Understanding-the-Nature-of-Poverty.aspx

The cycle of poverty is one, albeit major, factor but there is more that we must consider. The effects of the cycle of poverty do not apply equally to all impoverished demographics. In England for instance, those from impoverished Pakistani or Black Caribbean backgrounds are still much more likely to succeed in school than those from white working class backgrounds. Furthermore, the Cycle of poverty does not explain the lack of women in STEM or the lower levels of pay for skilled professionals from disadvantaged groups. Research shows that part of the issue here is that a lack of diversity is self-fulfilling. Those from underrepresented groups are less likely to enter fields where their group is underrepresented because they can see that their group is underrepresented:

This type of problem can only ever be resolved with active support from those in the fields that demonstrate a lack of diversity.

Beyond economic issues, we move onto the social and cultural issues. These are much more nuanced, varying from one group to the next. However, there are some common points that should be known:

First of all, discrimination STILL exists and it is STILL a problem, despite all the work to correct it. As many as a third of the British public class themselves as racist:
- http://www.natcen.ac.uk/blog/can-we-really-measure-racial-prejudice

Evidence from research shows that employers are still guilty of discriminatory hiring practices:

Many from ethnic minorities still encounter racist bullying in the workplace:
- http://race.bitc.org.uk/all-resources/research-articles/race-work-report

Recent research shows that well educated, well trained people from disadvantaged groups struggle for equality due to cultural segregation:
- http://www.lse.ac.uk/researchAndExpertise/researchHighlights/socialPolicy/EthnicDiversity.aspx

As already mentioned, I still frequently hear the argument for natural selection. The concept is by its very definition discriminatory. If you dig deeper I’m sure you will find that this argument is always a fallacy. An example of a common form of argument along these lines revolves around the overrepresentation of African Americans in some sports (and thus the underrepresentation of white Americans in those sports, I have seen both arguments). What you are seeing is in fact further evidence of the disadvantage African Americans face in education.
- http://www.sheysmith.com/2011/05/10/black-overrepresentation-in-sports-and-its-relation-to-education/

Discrimination also applies to groups identified by social status. Take the following article from a major English ‘Newspaper’ as an example:
- http://www.dailymail.co.uk/debate/article-1214549/Feral-youths-How-generation-violent-illiterate-young-men-living-outside-boundaries-civilised-society.htmlThe fact that such an article with such clear discriminatory language can even be published in England today is testament to the issues those from disadvantaged groups face… To quote “His life is already wasted”, how can the disenfranchised ever recover from that kind of societal attitude? It is nothing short of ostracism and it seems it is acceptable to many.

A study from the Runnymede Trust found that the White Working Class are discriminated against for a range of factors from their names to their accents and the way they dress:
- http://www.runnymedetrust.org/uploads/publications/pdfs/WhoCaresAboutTheWhiteWorkingClass-2009.pdf
- And for the Brits amongst us, think of the last time you heard the word ‘Chav’ casually banded about. Think about what that word actually means to someone from a working class background, and the effect that such an attitude will have on a wide group of people.

Within disadvantaged groups, expectations and behaviour change, a culture of discrimination can develop, resulting in low aspirations and even a social pressure to not-succeed. Furthermore, discriminatory attitudes from wider society results in underachievement of those from disadvantaged groups:

Discrimination is not something that those who experience it should just be able to shrug off. Regularly Experiencing discrimination causes not only mental but also physical health issues. These issues range from depression and low self-esteem to high blood pressure and respiratory illness:
- http://www.counselling-directory.org.uk/abusestats.html

I hope I don’t even need to link to any evidence of sexism on social media but it is worth re-iterating that people from disadvantaged groups who speak out against discrimination are often cut down by a torrent of abuse and threats of violence. Such people have had to move home due to the threats they have received. See ‘Gamergate’ for a starting point on this topic:
- https://en.wikipedia.org/wiki/Gamergate_controversy

Not a month goes by where I don’t hear of someone having to pluck up the courage to tell their story of the harassment they receive for being a woman or being a different colour:
- https://medium.com/@amelielamont/not-a-black-chair-8a8e7e2b9140#.gf85a0awk
- http://mobile.nytimes.com/2016/03/06/opinion/sunday/she-wanted-to-do-her-research-he-wanted-to-talk-feelings.html

The majority of women in STEM education face a ‘predictable’ stream of sexual harassment, intimidation and isolation, many are driven out of STEM by this:
- http://www.oswego.edu/Documents/STEM/8.3a_Brainard_Six-Year_Longitudinal_Study_Undergraduate_Women_Engineering_Science_1998.pdf

Culture, gender bias and sexism constantly works against women in STEM fields:
- http://www.theguardian.com/women-in-leadership/women-leadership-blog/2014/oct/20/women-science-engineering-under-representation
- And: http://www.usnews.com/news/articles/2015/10/21/women-still-underrepresented-in-stem-fields
Apparently it is acceptable to see a female workforce as harmful:
- http://www.dailymail.co.uk/debate/article-2532461/Why-having-women-doctors-hurting-NHS-A-provovcative-powerful-argument-leading-surgeon.html
- I appreciated this reasonable response: http://www.hrmagazine.co.uk/article-details/dean-royles-some-people-are-women-get-over-it

Apologies for the Wikipedia links but this is a very good starting point for reading up on the issues surrounding sexism in the workplace in the UK:
- https://en.wikipedia.org/wiki/Gender_inequality_in_the_United_Kingdom#Issues

In conclusion

One thing I have found during my time reading this information to present here is that I underestimated the sheer scale of the problems faced by those from underrepresented groups. This I find concerning as I was already aware of issues here and a need to actively work to overcome them. I had clearly underestimated the size of the problem that we all face in overcoming inequality.

There are clear and real factors working against those from disadvantaged groups that has resulted in an imbalance in the diversity of the software development workforce. Certainly it is not simply a case of saying “Hey, you are equal in the eyes of the law so sort yourselves out”. If we do not work to correct the imbalance then these pressures will continue to impede on equality. Those from disadvantaged groups face complex and wide-ranging challenges that clearly have a significant, damaging effect on their ability to compete for a successful software development career. If we as software proffessionals don’t act to level the playing field then not only are we maintaining an unfair world but, as many studies have shown, we are missing out on the benefits that a more diverse workforce brings.

P.S. An anecdote

Anecdotally speaking, there is currently a considerable shortage of IT talent, at least in the region of the UK where I reside and I also hear similar noises coming from across the UK and farther afield, over the pond in North America (http://techcrunch.com/2016/03/04/computer-science-is-the-key-to-americas-skills-crisis/). In my region, this shortage is so significant that businesses I know of are adjusting their hiring strategies from on-demand to ongoing. This is not only a problem to businesses but to the STEM industries as a whole as demand on their output is only going to grow. Surely anything we do to encourage more talent in the industry is a good thing for our economy and our society! I’d be interested to hear your thoughts…

Crash debugging Windows 10 Mobile UWP apps

January 27, 2016January 27, 2016 Stephen Roughley.NET Native, Debugging, SOS.dll, UWP, WInDbg, Windows Store Apps, WinRT2 Comments

So your app is crashing

This post explains how to get the details of the root managed .NET exception of a crash in a Windows 10 UWP app, specifically on the Windows 10 Mobile ARM platform. Hopefully this post will save you from some of the pain that I endured and aid you in getting to the bottom of your crashing app. Also note that with some minor differences – that I shall include as best I can in this article – this should in theory also work for debugging any UWP store apps on the x86 and x64 Windows 10 platforms although I have not tested this.

I’ll not detail the complete end-to-end process here as it is varied and lengthy and the core of the process is covered in excellent detail in a two-part series of posts by Andrew Richards of the Microsoft NTDebugging blog: ‘Debugging a Windows 8.1 Store App Crash Dump‘ Part 1 and Part 2.

The issue I found with following that series of posts alone is they are missing some key information if you are working on the Windows 10 UWP platform. No surprise when you consider that they were intended for Windows 8.1 Store platform. But they are full of essential details for the similar parts of the process on Windows 10 UWP and they got me so close!

In this post I will detail the information that is not already available in the above posts and how it fits into the overall process of debugging crash dumps from UWP apps running on the Windows 10 Mobile platform.

Enable and collect crash dumps

First off, make sure that Windows 10 Mobile will collect crash dumps by heading to Settings -> Update & Security -> For developers and ensure that the value of the settings labelled ‘Save this many crash dumps’ is greater than 0. I’d recommend at least 3.

Now reproduce the crash a couple of times to generate the crash dumps. The dump files should now be available under your device’s \Documents\Debug directory on the device storage. Note that it can take a few minutes to completely save the dump files and if you see any files here named ‘SOMETHING.part’ then the dumps are still being saved so come back in a minute or two. Move the dump files onto the machine where the debugging will take place.

Now on to the experts

Now I’ll pass you over to the aforementioned articles which explain how to fire up the dump files in WinDbg. Just as a heads up that at the time of writing this the latest Windows 10 Debugging Tools (WinDbg) are available from here.

If your crashes are indeed caused by managed code that you have written, generated or otherwise included then you will inevitably end up being directed to use SOS to elicit the details of the exception that is being thrown. This is where things got tricky for me and if you do get to this point then return here and read on…

Filling in the gaps

Now that you may have tried invoking, loading and even locating SOS and the CLR or DAC modules, I can tell you that these components are not where or even what the article describes. First of all I spent some time trying to confirm that the CLR or DAC was loaded as it should be according to most sources on this subject. Eventually after much trial and error I tried issuing a reload command to ensure the correct core framework was loaded. This is done with the following command (see documentation here). Also note that this step might not be necessary for you.

.cordll -ve -u -l

Which, for me, results in the following output:

CLRDLL: Unable to find 'mrt100dac.dll' on the path
Automatically loaded SOS Extension
CLRDLL: Loaded DLL c:\symbols\mrt100dac_winarm_x86.dll\561408BF43000\mrt100dac_winarm_x86.dll
CLR DLL status: Loaded DLL c:\symbols\mrt100dac_winarm_x86.dll\561408BF43000\mrt100dac_winarm_x86.dll

This is all fine but is not quite what I expected to see and leads onto the issue with using SOS. As you can see, SOS is supposedly loaded by the above command but normal SOS commands/invocations still will not work. The above DLLs gave me some clue to what was going on here and when I looked to see where this mrt100dac_winarm_x86.dl was located it led me to find the SOS DLL. In my environment, everything can be found here: C:\Program Files (x86)\MSBuild\Microsoft\.NetNative\arm and I can see here a DLL named mrt100sos.dll and a few variants thereof. So it looks as if there is a special distribution of SOS for the Universal platform, which makes sense.

NOTE: This is where the differences between the different platforms (ARM, x86, x64) will come into play. I suspect that this should be the same process for debugging UWP apps on all platforms but I cannot say for certain. At the very least you will see different modules/DLLs listed above and the different platform modules can all be found under: %Program Files%\MSBuild\Microsoft\.NetNative,

Armed with this knowledge I then headed back to Google and thankfully (luckily!) found one mention of using mrt100sos on a blurb for a non-existent Channel 9 show:

…This is very similar to how CLR Exceptions are discovered. Instead of using SOS, MRT uses mrt100sos.dll or mrt100sos_x86.dll (depending on the target). The command is !mrt100sos.pe -ccw <nested exception> . The same command(s) for CLR Exceptions is !sos.dumpccw <addr> –> !sos.pe <managed object address>.

And sure enough if you follow on from Andrew’s Windows 8.1 Store App articles with the above commands you will be able to see your managed exception in all its detailed beauty. In the following example <Exception Address> would be the value of ExceptionAddress or NestedException in your WinDbg output:

!mrt100sos.pe -ccw <Exception Address>

As an example, I had the following WinDbg output:

0:005> dt -a1 031df3c8 combase!_STOWED_EXCEPTION_INFORMATION_V2*
[0] @ 031df3c8
---------------------------------------------
0x008c2f04
+0x000 Header           : _STOWED_EXCEPTION_INFORMATION_HEADER
+0x008 ResultCode       : 80131509
+0x00c ExceptionForm    : 0y01
+0x00c ThreadId         : 0y000000000000000000001111001100 (0x3cc)
+0x010 ExceptionAddress : 0x7778afbb Void
+0x014 StackTraceWordSize : 4
+0x018 StackTraceWords  : 0x19
+0x01c StackTrace       : 0x008c67e8 Void
+0x010 ErrorText        : 0x7778afbb  "¨滰???"
+0x020 NestedExceptionType : 0x314f454c
+0x024 NestedException  : 0x008d09a0 Void

Taking the above NestedException address, I end up with the following command and resulting output. And this was all I needed to locate the bug.

0:005> !mrt100sos.pe -ccw 0x008d09a0
Exception object: 00f132ac
Exception type:   System.InvalidOperationException
Message:          NoMatch
InnerException:   <none>
StackTrace (generated):
IP       Function
65c279d5 ProblemApp_65810000!$51_System::Linq::Enumerable.First<System.__Canon>+0x99
65c27729 ProblemApp_65810000!$2_ProblemApp::Utilities::AssetsCache::<loadImage>d__4.MoveNext+0xa5
00000001
65539115 SharedLibrary!System::Runtime::ExceptionServices::ExceptionDispatchInfo.Throw+0x19
65539317 SharedLibrary!$13_System::Runtime::CompilerServices::TaskAwaiter.ThrowForNonSuccess+0x4b
655392c5 SharedLibrary!$13_System::Runtime::CompilerServices::TaskAwaiter.HandleNonSuccessAndDebuggerNotification+0x41
6553927d SharedLibrary!$13_System::Runtime::CompilerServices::TaskAwaiter.ValidateEnd+0x19
654ccea1 SharedLibrary!$13_System::Runtime::CompilerServices::TaskAwaiter$1<System::__Canon>.GetResult+0x11
65cf0285 ProblemApp_65810000!$2_ProblemApp::Utilities::AssetsCache::<Initialize>d__2.MoveNext+0x175
…

Would love to RTFM!

So hopefully this will help some poor souls who like me have to debug crashing Windows 10 Mobile UWP apps. If anyone knows of some proper documentation for the mrt100sos commands I would be eternally grateful!

LinkedIn Error “There was a problem sharing your update. Please try again”.

January 24, 2016 Stephen Roughley19 Comments

Obscure Error

I was trying to reply to a comment on an article I posted to LinkedIn the other day and kept hitting the error “There was a problem sharing your update. Please try again”. Just a note to help anyone who might come across this error when attempting to post an update to LinkedIn, there is an unadvertised comment character limit of 800 characters.

A little help?

It would be great if this was made obvious somewhere such as in the error itself or at least somewhere on the site but even searching the Internet for “There was a problem sharing your update. Please try again” didn’t turn up much for me. It wasn’t until I opened a support ticket that I was given this info.

I hope posting this here will at some point in the future save someone from wasting the time I did.

Things I wish I knew 10 years ago: Abstractions

January 18, 2016January 19, 2016 Stephen Roughley.NET, C#, Dependency Injection, Dependency Inversion, Design, IoC, MVC, Patterns and Practices1 Comment

We need to talk about abstractions

The main reason I decided to start this blog is that I have begun working for a company that has genuinely challenged many of my assumptions about how software should be developed. I have spent much of my career learning from the more prominent voices in software development about how to write software effectively. I have learned, practiced and preached the tenets of clean code, TDD, layered design, SOLID, to name a few of the better known programming practices and had always believed that I was on a true path to robust, maintainable software. Now I find myself in a position where over the space of just one year I have already questioned many of the practices I had learned and taught in the preceding decade.

I hope to share on this blog much of what I have discovered of late but for my first entry discussing programming practices I want to talk about abstractions. In particular I want to call into question what I have come to understand as overuse of abstractions – hiding implementations away in layers/packages, behind interfaces, using IoC and dependency inversion – as often encountered in the C#/.NET and Java world.

Abstractions?

I have been wondering lately if I have simply spent years misunderstanding and misapplying abstractions, but I have seen enough code written by others in books, tutorials, blogs, sample code and more diagrams than I can bear to know that I have not been alone in my practices. Furthermore, I have found myself on a few occasions of late in discussions with developers of similar experience who have come to share a similar feeling towards abstractions.

The all too familiar layer diagram. © Microsoft. https://msdn.microsoft.com/en-us/library/ff648105.aspx

So what do I mean by abstractions and what is the point of them, really? The old premise and the one that I would always reiterate is that abstractions help enforce separation of concerns (SoC) by isolating implementation details from calling code. The reasoning being that code of one concern should be able to change without affecting the code dealing with other concerns, supposedly because code dealing with one concern will change for different reasons and at different times than the code dealing with other concerns. Of course we mustn’t forget that one of the more natural causes of abstractions is the isolation of logic to enable Unit Testing. Ultimately the result is that software is written in such a way that the different code dealing with different concerns is kept separate by abstractions such as interfaces and layers while making use of IoC and Dependency Injection to wire the abstractions together. Furthermore it is worth me stating that the usual separate ‘concerns’ touted by such advocacy frequently includes Presentation/UI, Service/Application Logic, Business Logic, Data Access Logic, Security, Logging, etc.

[Authorize]
public class StudentController : Controller
{

    private readonly IStudentRepository _repository;
    private readonly IStudentService _service;
    private readonly IUnitOfWork _unitOfWork;

    public StudentController
    (
        IStudentRepository repository, 
        IStudentService service, 
        IUnitOfWork unitOfWork
    )
    {
        _repository = repository;
        _service = service;
        _unitOfWork = unitOfWork;
    }

    public ActionResult UpdateStudentDetails(StudentDetailsViewModel model)
    {
        if (ModelState.IsValid)
        {
            var student = _repository.Get(model.StudentId);

            student.Forename = model.Forename;
            student.Surname = model.Surname;
            student.Urn = model.Urn;

            _service.UpdateStudentDetails(student);

            _unitOfWork.Commit();
        }

        return View(model);
    }
}

Abstracted code, obscurity through indirection.

YAGNI!

I am not about to start claiming that everything should just be thrown together in one Big Ball of Mud. I still feel that SoC certainly is worth following but it can be effectively achieved by applying simple encapsulation, such as putting more repetitive and complex logic of one concern within its own class so that it may be repeatedly invoked by code dealing with other concerns. An example of this would be the code to take an entity key, fetch and materialize the correlating entity from a data store and return it to the caller. This would be well served in a method of a repository class that can be called by code that simply needs the entity. Of course packages/libraries also have their place, in sharing logic across multiple applications or solutions.

Where I see problems starting to arise is when, for example, the aforementioned repository is hidden behind an interface, likely in a separate layer/package/library and dynamically loaded by an IoC infrastructure at runtime. Let’s not pull any punches here, this practice is hiding significant swathes of software behind a dynamic infrastructure which is only resolved at runtime. With the exception of some very specific cases, I see this practice as overused, unnecessarily complex and lacking in the obvious transparency that code must feature to be truly maintainable. The problem is further compounded by the common definition of the separate concerns and layers themselves. Experience has shown me that when coming to maintain an application that makes use of all of these practices you end up with a voice screaming in your head “Get the hell out of my way!”. The abstractions don’t seem to help like they promise and all of their complexity just creates so much overhead that slows down debugging and impedes changes of any significant proportion.

how do we get the word out to stop creating layered project architectures like this? http://t.co/StXZxy5TCs

— Jimmy Bogard (@jbogard) September 28, 2015

With one exception I have never spoken to anyone who has ever had to swap out an entire layer (i.e. UI, Services, Logic, Data Access, etc.) of their software. I’ve personally been involved in one project where it was required but it was a likely eventuality right from the start and so we were prepared for it. I have rarely seen an example of an implementation of an abstraction being swapped or otherwise significantly altered that did not affect its dependents, regardless of the abstraction. Whenever I have seen large changes made to software it very rarely involves ripping out an entire horizontal layer, tier or storage mechanism. Rather it will frequently involve ripping out or refactoring right across all layers affecting in one change the storage tables, the objects and logic that rely on those tables and the UI or API that relies on those objects and logic. More often than not large changes are made to a single business feature across the entire vertical stack, not a single conceptual technical layer and so it stands to reason that should anything need separating to minimise the impact of changes it should be the features not the technical concerns.

Invest in reality

So my main lesson here is that: The reality of enforcing abstractions through layering and IoC is very different from the theory and usually is not worth it, certainly when used to separate the typical software layers. With the exception of cases such as a component/plug-in design I am now completely convinced that the likelihood of layered abstractions and IoC ever paying off is so small it just isn’t worth the effect that these abstractions have on the immediate maintainability of code. It makes sense in my experience not to focus on abstracting code into horizontal layers and wiring it all up with IoC but to put that focus into building features in vertical slices, with each slice organised into namespaces/folders within the same project (think MVC areas and to a lesser extent the DDD Bounded Context). Spend the effort saved by this simplification keeping the code within the slices clear, cohesive and transparent so that it is easy for someone else to come along, understand and debug. I’d even go so far as to try to keep these slices loosely dependent on each other – but not to the point that you make the code less readable, i.e. don’t just switch hard abstractions of layers into hard abstractions of slices. I don’t want to offend anyone, I’m just putting my experience out there… why not give this a try… I promise you probably won’t die.

Take a look at the following updated controller action. You know almost exactly what it is doing just by looking at it this one method. This contains ALL of the logic that is executed by the action and to anyone first approaching this code they can be confident in their understanding of the logic without having to dig through class libraries and IoC configuration. Any changes that are made to the action would simply be made here and in the DB project, so much more maintainable! Being completely honest, even recently, seeing code written like this would rub me up the wrong way so I understand if this gets some others on edge but I’ve come full circle now and am pretty convinced of the simplified approach. And its this dichotomy I’d like to discuss.

[Authorize]
public class StudentsController : Controller
{
    public ActionResult UpdateStudentDetails(StudentDetailsViewModel model)
    {
        if (ModelState.IsValid)
        {
            using (var context = new StudentsContext())
            {
                var student = context.Students.Single(s => s.Id == model.StudentId);

                student.Forename = model.Forename;
                student.Surname = model.Surname;
                student.Urn = model.Urn;

                SendStudentDetailsConfirmationEmail(student);

                context.SaveChanges();
            }
        }

        return View(model);
    }

    private void SendStudentDetailsConfirmationEmail(Student student)
    {
        ...
    }
}

Transparent, maintainable, intention-revealing code and no need for IoC!

This is just an opening

So this has been my first attempt to open up some conversation around the use of abstractions in software. I’ve tried to keep it brief and in doing so I’ve only just scratched the surface of what I have learned and what I have to share. There is still so much more for me to cover regarding what I and others I know in the community have been experiencing in recent years: Should we abstract anything at all? What is maintainable if not SoC via IoC? How do we handle external systems integration? What about handling different clients sharing logic and data (UI, API, etc.)? How does this impact self/unit-testing code? When should we go the whole hog and abstract into physical tiers? I could go on… So I intend to write further on this subject in the coming weeks and in the meantime it would be great to hear if anyone has any thoughts on this, good or bad! So drop me a line and keep checking back for further posts.

A C#/.NET SCEP Client

September 22, 2015September 23, 2015 Stephen Roughley.NET, ASN.1, C#, DER, PKCS, PKI, SCEP, SCEP Client4 Comments

Source Code: Download Zip Archive

As you have ended up here I must assume that, like me, you have a requirement to interface with a SCEP server using .NET and you have struggled to find any helpful examples of how to achieve this. I hope that this blog entry might be of some help to the few that venture into this obscure territory.

Background

As a little bit of background (there is too much to cover here), when I began my investigations into this area it seemed that there was pretty much only one option for implementing a .NET SCEP client and that was to use the Bouncycastle library, as the CLR System.Security.Cryptography API has some glaring omissions such as representation of a PKCS#10 request or ASN.1 DER type conversion. However this didn’t turn out to be quite the option that I had hoped. Firstly, I struggled to find any examples of a SCEP client implemented using Bouncycastle. Perhaps more importantly using 3rd party libraries always come with a risk, unless you know the code inside-and-out you have to place some trust in the authors to do the right thing. When it comes to integrating with a PKI – a major part of the infrastructure that you use to secure your digital estate – that risk seemed too great and so we had two options: either audit the Bouncycastle code-base or roll our own using the stock CLR APIs. Upon initial investigation of Bouncycastle I discovered quite quickly that it is a huge code-base and so I quickly turned to investigating building our own client, after all Bouncycastle did it so I knew it was possible, how hard could it be?

As it turns out it isn’t that hard, at least once you know the problem space, which turns out to be quite a mountain to climb for the uninitiated such as myself. My hope is that, having investigated the problem space and tackled some of the major hurdles, I am in a position to help some other people out in this area and provide a shortcut worth several days’ of effort that I could have done without.

The Example

To be clear, this isn’t going to be a full introduction into the ASN.1 and PKCS standards, there is plenty of that already very easily accessible on the Internet. Rather this is a simple working example of using the System.Security.Cryptography APIs to interface with a SCEP server. The code is documented with inline and XML comments that should point you to the online reference docs needed to understand the what and the why.

The Certificate Requests

The example I’ve provided here is in the form of a VS 2013 Unit Test project (converted from VS 2010 so beware of the old school Unit Test config!). This PoC was created for the specific case of receiving PKCS#10 signing requests and forwarding them to a PKI for enrolment via SCEP. This means that we were not concerned with creating the requests programmatically. If that is what you are after then I would suggest looking at using CertEnrol or automating the OpenSSL API. There are plenty of examples of using CertEnrol via the CERTCLIENTLib and CERTENROLLib DLLs, which are available on all Windows PC OS versions since, I think, XP. A good one is here: http://blogs.msdn.com/b/alejacma/archive/2008/09/05/how-to-create-a-certificate-request-with-certenroll-and-net-c.aspx?PageIndex=3. I struggled to find a simple way of forming a PKCS#10 request with the challenge password required by most SCEP environments. At any rate, I’m wandering outside the scope of this post and I ended up creating PKCS#10 requests for my tests using Open SSL at the command line.

The SCEP Server

When developing this PoC I was working with the NDES MSCEP component that comes with Active Directory Certificate Services. There are numerous other SCEP servers available, such as OpenSSL and theoretically this PoC should work with all of them. I just felt it was worth pointing this out so that you can bear it in mind while working with the example .

Lets Get Going Already!

You will need to tweak some of the example’s settings to fit within your environment but the logic itself is sound. Start by configuring URL of the SCEP server and the subject name that is used to locate the local/client certificate for signing the SCEP enrolment request. These can both be found at the top of the Unit Test class.

The example project contains a folder named ‘SCEP Tools’ that contains the Open SSL config and batch commands to create the PKCS#10 requests needed for the tests. Once created, update the respective request string variables in each test method with the request contents and you should be good to go!