Enterprise KM (Metrics)

1975_schwinn_7

Riding the Bike

As a child learning how to ride a bike I had both my uncle and my grandfather take me out to teach me.

Uncle Mark’s Approach

My uncle took off one training wheel and held the bike from the seat as he walked behind me on the path holding the seat.   He was looking to protect me and make sure that I didn’t fall.  His approach was also very logical.   I am sure that he thought about the process of taking one training wheel off at a time and that one wheel would allow me to learn how to offset the void of the other.   Before we got on the path, he explained to me what I needed to do.  I remember focusing on the pedals and trying to lean on the side that had the training wheel.   I fell and dropped that bike a number of times.   As I recall, it got to the point where he told me to keep practicing and we called it a day.

Grandpa’s Approach

Screen Shot 2013-12-15 at 9.33.24 AM

Now this memory is a killer for me.  Thinking back on this I actually remember how I felt in the moment.  The apartment building I grew up in is on the right.   We walked out the door and he told me to get on the bike. He had already removed both training wheels.  He had his hand on the back of the seat to hold me up and he said “when I push you, just pedal and look at the door in front of you”   In the picture above you see the two buildings almost facing each other.   You can see the entrance door from one building to the other, he was pointing at the door for the building on the left.   

He said “PEDAL”!! and he pushed me.   I looked up at the door, it was where he told me to go and I started to pedal in an instant and I mean an INSTANT, I got it.   I had mastered balance and I was riding.  It was one of those kick pedal bikes where you just push back to brake.  I had hit the brakes and spun the bike around like I knew what I was doing.  I turned the bike around and rode toward him elated.  Both of us were so gleaming and from that point on I knew how to ride.

Outcomes over Activities

Grandpa knew that to get the job done that he had to have me focus on where I was going and that the composition of activities involved were a subtle and covert process.   When I tried to focus on the individual activities of pedaling or taking on training wheels off one at a time, it was more of a distraction than helpful.   The result of having me focus on activities caused me to crash.   When my grandfather relieved me of that burden and helped me focus on the OUTCOME  I was able to learn to ride.    Of course there was room for improvement and there was still a lot for me to learn, but I was able to get where I was going right away.

Measuring the Enterprise

There are activities and business process to measure.   Most organizations aren’t in a position to care or understand the results when it comes to knowledge management.   MOST have yet to deal with the “Eleven Deadliest Sins of Knowledge Management .”

Eleven Deadliest Sins of Knowledge Management:

1. Not developing a working definition of knowledge.
2. Emphasizing knowledge stock to the detriment of knowledge flow.
3. Viewing knowledge as existing predominantly outside the heads of individuals.
4. Not understanding that a fundamental intermediate purpose of managing knowledge is to create shared context.
5. Paying little heed to the role and importance of tacit knowledge.
6. Disentangling knowledge from its uses.
7. Downplaying thinking and reasoning.
8. Focusing on the past and the present and not the future.
9. Failing to recognize the importance of experimentation.
10. Substituting technology contact for human interface.
11. Seeking to develop direct measures of knowledge.

Source: (Fahey & Prusak, 1998).

For some I have realized that no matter what I say, there are two truths.

1)People need to see SHINY OBJECTS —->Tools (i.e. Sharepoint, Wiki, Jive, Confluence etc)

2)People need to measure things GETTING TO THE SPECIFICALLY WRONG ANSWER WITH GREAT PRECISION.

If you are still reading..  and you want to learn more about the measures / metrics and you don’t care about anything else I have said here, you are in luck!

A_Practical_Framework_for_SharePoint_Metrics –Thanks Susan Hanley

KMmetricsguide–Thanks Department of the Navy CIO 2001 !~

Outcome to Measure 

Ok.. now that we have fed that beast..  Really,  you really need to think about a few things for the knowledge ecosystem relative to the business.

How can we Increase Revenue?

How can we Improve Productivity ?

How can we Reduce Costs (overall)?

If you want to break these down to areas like operational resilience and continuity or knowledge transfer or other areas, you can!  The bottom line is how is what you are doing in your knowledge practice going to help your business  / organization.   Even the not for profits have desire their work to be more than naught.

If you find yourself focusing on the tools or the measures / metrics, “results aka (s*&%) happens”

“It boiled down to courage and tenacity”: My “Inbox Interview” with Howard Cohen, Community Manager at DISA Forge.mil and Technologist by Chris Maher

I was interviewed a few weeks back by Chris Maher on Linkedin.  The topic was concerning “Trusted Computing”  and ramblings on security.

CM: Howard, as you know, I quoted you at the 2011 NSA Trusted Computing Conference & Exposition: “Well.. I believe in Americans. I believe that when we see various challenges that we individually step up and out to deal with them. We have put your faith and trust in leadership and leadership has been pounded with more work than they can handle (yes, I am being nice). That being said, it is up to us individually to lead where we are. We must individually work to change our own behavior and look to influence others by leading from where we are. If I am a Janitor, then I look for ways to be efficient in cleaning and thrifty in spending for supplies, or find ways to reuse supplies. If you are an Executive Assistant, find ways to make a difference in the office. If you are a Technical Strategist, teach everyone everything you know about Service Orientation and Trusted Computing and technical reuse models. It doesn’t matter who you are, it matters what you do. Our jobs do not define us holistically. In recent days I have seen civilian leaders (you know who you are) step up to the plate and take risks in order to share their ideas on how to create a more effective and efficient acquisition solutions. It isn’t only up to them. We will find more success together by working to change these behaviors and tackling the challenges we can see one person and one problem at a time…” (SOURCE):https://cohenovate.wordpress.com/category/howard-cohen/

It’s a great quote for a variety of reasons. That said, I want to focus on your awareness of and experiences with Trusted Computing. How you were first introduced to Trusted Computing?

Chris,  Thank you very for clearly understanding and articulating the message of “leading from where we are.”  I have been working for the Department of Defense for close to a decade now, before that I worked at a school division and the commercial industry.  I have worked for Joint Forces Command, Joint Staff and now DISA.   I started hearing about Trusted Computing while working at the school division, if anyone is going to break your system it will be the kids.  I learned a great deal about system hardening as I entered the world of military architectures at J8.   I started at US Joint Forces Command by using security technical implementation guides (STIGs) as we call them.   Prior to that I was using non-military oriented technologies like hard drive sheriff, deep freeze, bootable cd os (barts PE), stuff like that. 
And, in your estimation, why does Trusted Computing matter? Why is it important?
In enterprise computing you want to be able to leverage standards. We need the ability to look at metrics and we need to understand what “expected behavior” is.  In other words, we need to be able to know when something is not working right.   So you need standards so that experts can be on the same page and understand what they are looking for as “normal” as opposed to seeing something that “interesting” , if everyone is doing their own thing at the enterprise it makes it very complicated to know what the heck is going on.   You have “shadow IT” that will compromise the integrity of the network simply because it exists.   When working in an enterprise users and operators need to trust that mechanisms are in place to protect them.  I can go on about this but the bottom line is that to know if something is wrong you need to establish that something is right.  I believe that is why Trusted Computing is important. 

CM: As you may know, Richard Stallman once rebranded Trusted Computing (TC) as “Treacherous Computing” which made a neutral set of technologies out to be a threat to open computing and/or our civil liberties. Stallman conflated Microsoft’s Palladium effort with the word of then TCPA. Ever since, TC has been dogged by the adjective “controversial.” For me, TC (including self-encrypting drives) actually protects my civil liberties by arming me, the digital citizen, with technologies that can defend my information from any intruder… including an intrusive government. But that’s just my opinion. How do you assess the intersection of Trusted Computing and civil liberties.

As long as there are people involved in computing, there are going to be hackers.   As long as we are at war with others, there will be people who will look to harm us in the real world or through technology.  Sure you are sharing the standards but I would say process and method are two different things.  In other words, you may have common technological frameworks and standards but how enterprise strategists think about and employ these technologies are different.   For example, I know of an organization that uses two layers of username and password and additionally requires a common access card, all of which are standardized.   The practice is abnormal but if a technologist was brought in to help solve a problem once he or she understood the architecture and because they are using standardized technologies and platforms they can help solve the problem.   I equate it to having a human in the loop.  People are your greatest protection mechanism as well as your greatest threat.   In terms of civil liberties, I think we have some problems with the law more than technology.  We don’t have a right to privacy, it isn’t guaranteed by the constitution and that means corporations and people are free to snoop around our business.  When that gets into information gathering and data aggregation it poses a much bigger problem than just technical mechanisms to protect our data.  It is more about what information did your city just put out about you and your home value, stuff like that.  So, in other words I am not sure that Trusted Computing makes a difference here unless we are just talking about me protecting my local hard drive.

CM: Much noise is made by IT professionals about the difficulties of using TC, specifically going into the BIOS and having to turn on TPMs. And it must be said that there has not been the development of many applications that leverage TPMs. In your experience, is Trusted Computing too hard to implement?
I have seen full disk encryption at the corporate level and while working with the government.  I have not seen BIOS based modules employed and I don’t have personal experience with BIOS based secure computing.  As I mentioned earlier, while working at the school division we used a device call hdd sheriff and some technology out of Israel to perform persistent drive management and encryption.   This was over 10 years ago too but the concepts have been around for a long time.   There aren’t a lot of commercial options that I have seen at the application level that use TPM’s but I think there is value there depending upon the requirement.   This is all about balance.  Risk is the key.  How much is this going to cost you?  What are the implications?   If I am working in the financial sector, I want as much technology as I can to protect my information.  The same could be said for the medical industry, I haven’t figured that one out yet but I am sure there is a good reason.  

CM: It’s been my contention that government MUST take the lead in adopting and recommending Trusted Computing. In this regard, I’ve been heartened by the NSA’s (more or less) full-throated endorsement of TC and by the CESG’s recommendation in favor its use. Further, as you may know, NIST 800-155 (in draft form) has recommended (or will recommend) the use of a hardware root of trust as a foundation for BIOS Integrity metrics. Still, it seems like .gov and .mil domains have been quite slow to fully adopt these open standards and technologies. In your view, what’s the state of play re: TC adoption within our government?
This is about cost of implementation and ability to implement.  In other words, as long as there are programs that are “Programs of Record” with Title 10 authority, essentially meaning that they can control their own technical destiny there won’t be adoption unless it becomes part of the culture.  For example, while working for Joint Forces Command I stood up one of if not the first accredited virtual infrastructure.   Most people were getting rejected at the time because hardening didn’t exist aside from the vendor best practices.   Information Assurance folks were afraid to take the risk, although it could mean millions in savings.   It boiled down to courage and tenacity.   The government leadership I worked with and for championed the idea and helped me bring people together by supporting our teams ideas.   It took many briefs and I think I have stock in some chocolate company now as well to get people to believe that there was value in virtualizing the infrastructure.   I know that sounds funny now because so many have adopted virtual technologies.   Here is the kicker though, today even though virtualization has proven to be of great value there are many government programs that haven’t virtualized and / or won’t go because of requirements and title 10 authorities.  CM: A great deal of academic and industry research has focused on the value of TC when it comes to authenticating users in a cloud-computing context…as well as using TC to protect user’s data in the cloud from the “insider threat.” Speaking specifically about the cloud-computing context, how important do you think TC technologies (TPMs) and protocols are as enablers?

As I started working on enterprise computing concepts and strategies, I started to see a trend.  Thomas Erl talks about this in his service oriented architecture books but it has to do with understanding dependency.  Cloud computing may increase risk.  Notice I say “may” instead of will, the reason is that every enterprise situation and IT ecosystem is different, remember earlier when I was referring to process and method being two different things.   Regardless of the situation organizations will have dependencies, for example you need communication services to connect to the Internet.  As you increase services and connectivity requirements it is likely that you introduce more dependency.
The cloud really refers to “off premise” services. These services are interconnected enterprise services that go beyond an organizations local physical infrastructure.   This is very important to realize because it means that hardware and IT resources are still potentially under trusted controls of an organization which of course then leads to leveraging organizational standards etc.   

The difference is that when you have a dependency on a “cloud provider” that is outside of your organization you build dependencies in which you may lose control over the IT resources.  As you give up autonomy or operational governance, you become more reliant on legal remedies.  In other words, SLA’s or Service Level Agreements become critical to the organization.   This relates to Trusted Computing in a lot of ways, for example a service provider may need to employ certain (TPM’s) prior to an agreement of use.  This increases the cost to service providers and also may limit choices as to what service providers’ organizations can use.  An example is that Amazon offers Federal services with enhanced security.  I am not advocating for any service provider, I am simply saying that as cloud services increase, the costs of these services will increase and the demands of security and stability increase.   In the grand scheme of things it wasn’t that long ago that most folks were on dial-up, it was $9.95 to $19.95, today most people pay $40.00 for Internet services not including the extra services they pay for while on the Internet.  As these costs increase, it pushes the price of everything up, simple economics.  Trusted Computing in the cloud is costly, but organizations when moving to the cloud will need to absorb these costs.  

My key point is that we can’t rely on technology alone.  Technology as it is today can be overcome by the human brain.  That being said, we still must put barriers in place to slow down attackers enough so that we can identify in some manner that our information is being attacked.   It is the difference between having a lock on the door and adding a security system.   Some people would say that adding a security system adds no value or is a waste of time.  I think as we continue to build technological solutions to thwart attackers or secure the enterprise, we strongly need to consider how we can keep “a human in the loop” and have people involved in watching the various stores.   As we move forward with these kinds of discussions we truly need to consider people, process, methods and finally tools which in my mind is where a lot of the Trusted Computing area currently addresses.  

Happy 4th of July (SPECIAL DISPATCHES FROM THE FIELD)

Today is the 4th of July.   What does it mean to you?  Joni Douglas writes “Do we ever reflect on the hardship and terror that the early Americans must have lived through day after day? Do we truly understand the hopes and dreams that lay secretly hidden away in the hearts of the people back then?”

I appreciate our past, it is what defines who we are and what we will be.  Americans as a whole, are smart, bold, brave, courageous, tolerant and most of all… AMERICANS.  Meaning when times are tough we pull together and we take care of each other.   We have a long history of thinking on our feet and challenging oppression.  We are people as one but independent, we are a tapestry that is tied together by the concept that we are all created equal and that we have unalienable rights Life, Liberty and the Pursuit of Happiness.  

Freedom isn’t free and our rights that we as a people have declared through our “Declaration of Independence” come at a cost.  As an American, I served in the US Navy and frankly I was too young at the time to really understand these concepts.  As I have become older, I have learned that how truly lucky we are to live and be part of this nation.   What is unfortunate and clear is that we are distracted by entertainment and we aren’t paying attention to the wars that we are fighting RIGHT NOW today.

If you didn’t know, I will share this with you, we are constantly under attack.  Right now while you are potentially firing up your grill to make some dogs, men and women are working to protect us on every shore including our virtual borders (cyber).

THANK YOU to our service members.

What I am about to share with you is very special.  This comes from a man that I personally consider a living hero and true patriot. This is message that he sent me today July 4th, 2012 from Bagram, Afghanistan.   As long as he is sending me notes and allows me to post them, they will appear here on my blog.

Thank you Ken Williams!

“Dispatches from the Front:4 July 2012.”

They are provided to give you a bit of insight into what is happening in this hostile land.  It is hard to imagine that in ancient history, Bagram, Afghanistan was once the primary crossroads between the civilizations of India, East Asia, Central Asia, the Middle East and thence Europe.  Today, Bagram is the crossroads to hell.

Dispatches from the Front: 4 July 2012

Happy 4th of July and I wish for you and your family only the very best this world has to offer.

Today is a regular work day here.  That is fine by me for there is no Dairy Queen to cruise and there is much to be done.

This dispatch is to inform you on the strategic objectives of Regional Command – East (RC-E) and a few personal observations.  The following information is unclassified.

There are five operations objectives.

1.            Accelerate the Afghanistan National Security Force (ANSF)

capacity and the transfer of lead security responsibility to the ANSF.

2.            Improve security by, with, and through the ANSF.

3.            Support the Government of the Islamic Republic of Afghanistan

(GIRoA) in the development of sub-national institutions, civil service capacities, Rule of Law, and Socio-Economic Development initiatives and programs across the Coalition Joint Operating Area-Afghanistan (CJOA-A) and synchronize their efforts to support security objectives.

4.            Inform and influence the Afghan populace.

5.            Reset the Theater.

There is a Classified Campaign Plan that lays out what needs to be done.

It reads as a “tall order” from anyone’s perspective.  However, as I review the nightly situation reports, I am seeing the Afghanistan National Army (ANA) do more and more and they are successfully taking the fight to the enemy.  This may be out of necessity for I am sure the Taliban are eagerly awaiting our withdrawal.  We all realize that history is written by the victors, but honestly, the ANA is doing well.

There are still many hurdles to cross and acceptance of different ways

of doing business by both the CJTF and ANA.   I cannot state any

stronger, the real test of their effectiveness will be measured when there are no coalition forces in country available to  provide efficient and lethal support when needed.  I worry also about long term sustainment.

A real surprise to me is the obvious absence of the Afghanistan Mission Network here in RC-E.  It must be in Kabul and used by the International Joint Commission and ISAF HQ for it surely is not being used here.  The primary means of C2 by the commander of RC-E and his subordinate commanders is CENTRIXS ISAF at the SECRET level.  As I become more recognized in the JOC, I will find out for sure if the AMN extends here.

Personal observations:  Since the very beginning of my career, and now as I have grown old with white hair, one thing has always remained constant; the men and women of the U.S. military are our greatest National treasure.  What gives me hope is knowing they are a microcosm of our society proving the majority of our U.S society is still solid.

As hard as the liberal media tries, they have never been capable of tarnishing the trust most U.S citizens has in our men and women in uniform.  As long as Soldiers, Sailors, Airmen, Marines and Coast Guard men and women continue to display unbridled strength, ethics, morality, integrity, courage, duty, and honor, I fear not what the politics of this world brings forth. But I do fear that our politicians have lost sight of what it takes to keep this Nation secure in the freedoms so precious to all of us.  They are the ones who in my opinion have lost sight of their duty.  They should try to remember as Gen. R.E. Lee once said, “Do your duty in all things. You cannot to do more.  You should never do less.”

Sheets arrived yesterday and were immediately installed.  This morning I realized they do make a difference in the comfort of your rack.  Life is great!!

At 0900 local yesterday, I attended my second “Fallen Hero” ceremony.

This time it was a civilian killed in his bed asleep when his luck ran out.  A rocket attack on a different forward operating base (FOB), south of Bagram, ended his life.  Whether the man was a contractor or a GS, he was given the same respect and honors as given to a fallen military person.  I am not ashamed to say that tears once again fell down my face as the open HUMVEE carried him slowly pass me.  The stars of his flag seemed larger and the strips wider and brighter as the sun bore down on

us.   This time there were 5 musicians and I recognized one of the songs

as, “Amazing Grace”.  You are left humbled once the Sergeant Major dismisses the formation.  It is then you realize, “only for the grace of God, go I.”

Later, at 2000 local, I attended my third “Fallen Hero” ceremony for a young soldier killed in action against the enemy.  The moon was positioned perfectly over the center of the awaiting C-130.  It was full which made the bare mountains surrounding us more ominous and they seemed closer.  The wind during this time of year, at night, is constant and blows up to 20+ mph.  The color guard struggled to keep the flags under control.  Honors were rendered.  Silent tears fell as one more soldier was taken from the field of honor and sent on his way home to Arlington.

Semper Fidelis,

Ken

CJTF-1, ID, CJ5 Assessments

Task Force Defender

Bagram Air Field, Afghanistan

APO, AE 09354

Community Management Metrics

Community Metrics in Context

In late 2011 I started working for DISA on the Forge.Mil project.   Forge.Mil is a really great concept that was born out of the need for a change in the Department of Defense concerning software development and application life-cycle management.  Some the concepts that attracted me to Forge (that is what I call it), was that it integrates knowledge management, information management, software development and object relational mapping to authoritative data.   As a System Integrator, I would always have a challenge when it came to documentation and information management aside from required documents for information assurance.   Forge enables teams to share data in context of IT in ways that are similar to other knowledge systems but essentially designed for software development and software projects.

This is a unique and challenging situation because most of the time in social communities people have a desire to communicate and collaborate.   In this environment with hundreds of different projects and thousands of users motivators for the individual developer varies.   In other words, some kids may not want to play in the sandbox, while others excel at building castles together.    As Morgan Freeman says while discussing the universe “Answers are terminus, it is the questions that are where it’s at.”   With that said, I had and have a lot of questions about the community management concerning IT related work.

How do you know what you need to look at to determine what is happening in your community?

I have spoken to a lot of people about this in the past few weeks and I have scoured the internet looking for questions and answers to consolidate a nice list.   One of the first things to consider though is asking yourself about the purpose of your community.   Who are my community members? What does the population look like?  What do they do on a daily basis?  What makes them different from each other?  What makes them the same?  Questions like these really help flesh out what you need to measure.

I put together a document on metrics and it was really a mashup of data from a lot of community managers, blogs, discussion posts and some publications.   I am posting most of the content here so that if you happen to stumble along here, you can take what you need and put together your own thing.   I have links to most of the authoritative data sources but if you come across something you wrote and I didn’t source it properly (PLEASE) let me know and I will include the proper reference.  I am not looking to take credit for the hard work and thought of others.

Think -Wait-Think- Do..

“We need to first define the problem.
Albert Einstein once said:
“If I had an hour to save the world
I would spend 59 minutes defining the problem
and one minute finding solutions”
And I find in most organizations
people are running around spending sixty minutes
finding solutions to problems that don’t matter.”
~ Stephen Shapiro

The information included in this post is a primer to get you thinking about what you need to do in your organization.  These are not the answers, these are the basis for the questions.    Most technical people think about the answers as quickly as the problems present themselves.   As a “leader from where you are” it is your job to help keep the solutions at bay until you determine that enough questions have been asked.   Additionally, the process should never end.  We should always ask questions, tune, adjust, qualify, quantify and wonder.  If you want to talk about the technical side of this work contact me separately.

Context First

Just remember to always keep the qualitative metrics in mind first as you consider the quantitative metrics.   One of the first posts that I came across that I thought was very interesting located here talks about a table with some basic attributes of consideration.  I added some attributes in the paper I put together but I could have very well just stuck with these.

“It’s either quantifiable or it’s not measurable”

“It’s either quantifiable or it’s subjective

“We have to quantify it in order to measure it” – Dr. Mel Schnapper, PhD

Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force:

“There are three kinds of lies: lies, damned lies and statistics.” – Autobiography of Mark Twain

Program Managers must understand what the total cost of implementation is. How do we capture a baseline? What is the total cost of ownership? What is the total cost in investment? What are the estimated savings? How do we know that we are achieving our goals? How do we know that we have growth? How do we know that this project is sustainable? How do we know where to course correct? What are the measures of performance? What are the measures of effectiveness? How do we define “effective”? How are we measuring success?

Community Goal: Drive Reuse

It is vague to consider a generic goal of “reuse.” There are a number of community components that can be measured for their reuse-ability.

• How do we know if people are informed about new features?

• How do we offer training on these features?

• How do we know if people are using these new features? If not using – Why?

• Are they using other web sites? Are they linking to them?

• Are they using their tool as a SharePoint or as a code collaboration tool?

• What are linkages / commonalities between projects?

1:Pre-Built Components Reuse

• Goal: Reuse Pre-Built Components

• Question: How much interest is there in each component that is meant for reuse?

• Metrics: Number of downloads, number of posts on the discussion posts around the component

2: Code Snippet Reuse

• Goal: Reuse Code Snippets

• Question: How many different places do code snippets intended for reuse appear within the community?

• Metrics: number of projects where this code snippet is being used, and/or where it has been “forked”

3: Knowledge Reuse (non-source code)

• Goal: Community Members are sharing knowledge within the community.

• Question: How often are users looking for information among the entire community?

• Metric: number of attempts to search for information community-wide on a particular topic.

4: Expertise Reuse/Reputation Management

• Goal: Users are able to find community experts that can help or guide them.

• Question: How would a user find an expert on a specific topic?

• Metric: determine which are the most popular community-wide search topics -> determine which community members post the most information about the most searched topics.

5: Trust in Reuse

• Goal: Users within the community trust that the community is a place to find valuable, trustworthy answers

• Question: Do users believe that they can rely on the information they are getting outside of their silo?

• Metric: while this is probably best learned through surveys, the number of hits/interest around specific topics that are demonstrating a high level of activity can also provide evidence of success in reuse trustworthiness.

(https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.community-mgmt/wiki/DriveReuse)

Creating and Managing a Healthy Community

Growth

For growth, it is how many people are invested in what you’re doing that matters.

In terms of Growth, we look at:

• Twitter followers/Fan (LinkedIn, Milsuite, other) page members/social media friends.

• Blog Subscribers.

• # of Active commenters.

• Member registrations.

• Unique visitors.

• Ratio of posts to comments, types of comments.

• # of Message posts, if a forum.

• # of Conversations over a month period.

Presence

How visible are you in your space and how does your visibility measure up against that of other defense communities?

In terms of presence, we look at:

• Buzz over a 30 day period.

• Types comments/posts written about Forge.Mil – mentions (linked or unlinked).

• Who authored the mention – client, colleague, recognized social media contact, influencers, etc.

• Where was the mention located?

• How often does your community share your content?

Conversation

Presence is who’s talking about Forge.Mil, Conversation looks more at who Forge.Mil is talking to and the effectiveness of those conversations. Measuring the types of conversations Forge.Mil leadership and Community is having is an important metric because it shows leadership where your time is being spent and how people are engaging with you.

In terms of Conversation, we look at:

• Breakdown of the types of conversations being had– support-based, link sharing, friendly banter.

• Time spent on each conversation group. What’s more cost-effective – social media or phone/email, defense connect online chat?

• Whom you’re conversing with – customers, prospective customers, colleague, outsiders.

• Conversation spread and growth?

• Actionable knowledge learned about core audience.

Sentiment

More important than simply knowing Forge.Mil is being talked about knowing what people are saying about Forge.mil and how that’s changing over time.

In terms of Sentiment, we look at:

• Emergence of Evangelists – onsite and off.

• Ratio of positive/neutral/negative mentions (i.e. satisfaction).

• Forge users recommending the community, passing it on to friends.

• Frequency of community members responding to/helping other community members, overall “vibe” of the room based on tracked interactions.

• Community members defending Forge.Mil on negative blog posts and feedback.

Conversions

In terms of Conversions, we look at:

• Community member, followers, frequent blog commenter, etc).

• Customer loyalty forum (Charter) community conversions – how many times do they refer?

http://outspokenmedia.com/social-media/how-to-measure-community/ (great content)

Metrics as an indicator

A community must support business goals and the current (and prospective) community members themselves.

  • If the business goals are not defined, the community risks being feature-driven and may suffer from shiny-object syndrome.
  • If the community members are not involved in the success definition process, the community risks being irrelevant to its members.  (Community Charter)
  • If business goals are undefined, or if community members themselves are not involved in the definition of the community (it’s for them, after all), the community’s risk of failure grows substantially.

There are generally two types of metrics

Qualitative Connecting the dots can be challenging, since the points of data capture for qualitative metrics are often two or more degrees of separation from the data.  Qualitative information is contextual information derived by the relationships of data points.   Qualitative metrics takes variables, attributes, values and relationships into consideration in order to determine the likelihood of a condition.  Qualitative information / data is based on a deep understanding and the correlation of data.

Community example:

There are various interactions in the community that can be measured by frequency of visits or downloads but we may not know why.   This is an important factor as understanding the drivers can give us the ability to reinforce successful behaviors.

What does it mean? Interpretation of the data once captured.

Attraction -The ability to attract an initial audience.

Attention – The ability to ‘reel them in’ and have them go deeper into community content

Adoption – the ability to ‘convert’ them into Community users have them contribute to discussions.

When placing the aforementioned metrics into the 3 framework categories above there is a clearer understanding of what we are actually measuring:

Each of these has a quantitative component although it is difficult to measure external connectivity (meaning interactions that occur as a predecessor or successor as a result of the interaction in the community outside of the environment.

Quantitative Quantitative metrics are gathered directly through the observation and measurement of data.  There is a high degree of transparency and a direct correlation between action and outcome with quantitative metrics.

The data points can be classified as large Q for quantitative or small q for qualitative.

Attraction – The ability to attract an initial audience.

Attention – The ability to ‘reel them in’ and have them go deeper into community content

Adoption – the ability to ‘convert’ them into Community users have them contribute to discussions.

When placing the aforementioned metrics into the 3 framework categories above there is a clearer understanding of what we are actually measuring:

Attraction (Q,q) Attention (Q) Adoption (Q,q)
Visits Page views per visit User Creation
Unique Visitors Bounce Rate % Returning Visits
% New Visits Length of Time on Site Interactivity Rate (q)
% Users by service First time contributions Account age
% Users by organization Most active members Content ratings
% Users by agency Mentions by influencers New posts per month
% Users by unit Overall project activity Reputation changes
Page views overall Ratio: Views / Post Topic activity by project
Awareness (q) Ratio: Posts / Thread Individual project activity
Inbound Links Ratio: Searches / Post % Content with “tags”
    Innovation (e.g. # new product ideas sourced from community) (q)
    Member Satisfaction (q)

Other potential factors (some duplication)

Visitors Metrics: Such as Unique Visitors, Bounce Rate, Pages per Visit, Pageviews, Time on Site, Keywords, and Referring Sites

Members Metrics: Such as New Registrations, # of Active Members, Completed Profiles, Pages per Visit, Pageviews, and Time on Site

Contributors Metrics: Such as # of Edits, # of Comments, and # of New User Generated Content

Evangelists Metrics: Such as # of External Invitations, # of ShareThis external shares, # of Mentions on social media sites (e.g. Twitter)

Leaders Metrics: Such as # of Active Admins, and # of Active Moderators

http://blog.wiser.org/metrics-for-the-busy-community-manager/

Getting a Baseline

If we have 1,000 members in the Forge.Mil community and we do some basic analysis and learn that each week there are approximately 50 new threads posted in our project discussions and about 500 replies.

These three pieces of data are a good baseline for starting to look at ratios. A bit of quick division and your ratios come out like this:

Ratio of posts per member, per week: 0.05 (50/1000) (baseline)

Ratio of replies per member, per week: 0.5 (500/1000) (baseline)

Scenario 1 – Members Up But Engagement May Not Be Keeping Pace

Fast forward one month and let’s say our community has grown to 1,500 members. This is a huge increase, but are the ratios keeping pace with the growth? Let’s say, for example, that there are now 100 new threads posted (a lot of the new members introduced themselves) and about 650 replies each week.  Your new ratios look something like this:

Ratio of posts per member, per week: 0.0666 (100/1500) (increased)

Ratio of replies per member, per week: 0.4333 (650/1500) (decreased)

The ratio for posts per member (per week) has gone up thanks to all those new members introducing themselves to the forum. However, the ratio for replies per member has decreased and hasn’t kept pace with the growth. Using these findings as a starter, a quick look around the community may reveal that all those new members are introducing themselves but are not engaging or replying as much.

If you can not measure it, you can not improve it.
– Lord Kelvin, 1883

Scenario 2 – Members Down But Is The Quality Better?

In another example, let’s say community experiences a slight loss in membership, dropping down to 900 members, but the number of posts and replies stays the same. The new ratios are:

Ratio of posts per member, per week: 0.055 (50/900) (increased)

Ratio of replies per member, per week: 0.55 (500/900) (increased)

The ratios above show that community is actually looking very healthy, and so the decrease in membership is likely represented inactive members.

Ratios are a simple way to start putting context to data.  In the first example, the community numbers increased, which looks great, but the number of replies didn’t keep pace with the growth— leading to some actions or follow-up with the newer members. In example two, the overall numbers dropped, which looks negative on the surface, but the ratios show us that the community is actually healthier now.

(http://thecommunitymanager.com/communities-and-the-ratios-that-bring-insight)

Metrics Review: reviewing your performance at least every month, and compare month-over-month (MoM%) and year-over-Year (YoY%) growth rates in a spreadsheet

Other notes:

NOTES:

Here is some other relevant information.  This comes from some of the collabnet notes in their community wiki https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.community-mgmt/wiki/CommunityMetrics

Community Metrics should be

  • Based on agreed upon, communicated community goals
  • Centrally managed by the Community Manager and Internal Community PM
  • Communicated ‘up’ to stakeholders in a way that provides information about community goal progress, community health, and ROI
  • Communicated ‘down’ to the community in a way that promotes the community and identifies opportunities for community growth

Community Metrics should be planned to come to an agreement on

  • The various aspects of the goal: For example, the goal reuse can include: code component reuse, code snippet reuse, knowledge reuse. Each aspect of the goal is itself a sub-goal that will be further analyzed and measured differently
  • The questions around the goal: For example, the sub-goal of ‘code component reuse’ begs the question “How many times have reusable code components been reused”
  • The metrics to be collected regarding the goal: For example, code component reuse’ could be measured by collecting: # of times reusable code components have been downloaded
  • The methods employed to provide the goal results: For example, how/when will the metric/s data be captured
  • Should the metrics data be trended over time: For example, Should the metrics data be compared against other data for relevance and further analysis

Measuring Community Health

Measuring community health is not looking at traffic numbers or page views. A community is considered healthy when:

  • The community goals are being met, such as support questions are being answered
  • Attitudes in conversations are light and friendly
  • Conversation is two-way or more, but not just single posts
  • Issues are being resolved and needs are being met

Community health can NOT be determined by simple numbers. It takes a community manager, or several, to read through conversations.

A community is considered unhealthy when:

  • Community goals are not met
  • Conversations are non-existent
  • Flame wars and arguments are taking over the community
  • Community members are unhappy with the interaction, or lack there of

Type of messaging that reveals the health of a community:

  • Messages of appreciation
  • Messages with solutions to problems posed by community members
  • Requests for information are answered
  • Conversations are friendly and sometimes lengthy
  • Complaints are directly addressed by the community manager or other company staff
  • Community leaders emerge from the conversations

Good luck!

Keep reading and writing!  Keep asking questions and publish what you find, we can all use the help.  Remember people first, context and scope (qualitative) and the quantitative should help with the qualitative.    If there is an interest, I can post some information about tools but most of the focus is on people, process, and methods.. cheers!