1. Wireless access is at  HFA-Guest  /  <password listed on the fridge>  .. no the password is not “password listed on the fridge” .. don’t bitch about the password, it’s free WiFi bro!
2. Don’t adjust the temperature!  If you’re cold, put on a sweater, toque (beanie if you’re south of the 49th parallel).  If you’re hot, have a drink with ice, and if that doesn’t help, then piss off.
3. Austin tunes over-ride. Period. Don’t care you want to listen to some wimpy East Coast, West Coast, Popular rock or Northerner crap.
4. If there is a NFL game on, then the game is ON .. don’t expect much else.
5. Don’t suck up *all* the bandwidth in pr0n, dude. Really? I know it’s you. Remember what I do for a living?? Yes there are “proxies” on friend’s free Internet connections. Duh.
6. GPS (aka TomTom or Garvin) HIGHLY recommended for out-of-town-ers.
7. GPS (aka TomTom or Garvin) HIGHLY recommended for in-towners.
8. Be energy conscious. Rinse your damn dishes (don’t be lazy) – that’s what the drying rack is for. Duh.
9. You consume the last bottle of _______, REPLACE it. Damn, there is a Tarjay (Target for our American friends), or H-E-B, or Randall’s within walking distance!
10. Do NOT put your drinks on my Red’s Porch tab. Food is negotiable.
11. You MUST have a valid reason to go to another joint than Red’s
12. Yes, it IS a shower curtain rod like Marriott’s.  No, I didn’t steal it. Nice, eh? That’s another 5.5″ of room in the shower!!
13. No you don’t have to come run with me in the morning at 5am.  Nor do I .. but sometimes I’d appreciate the encouragement out of bed.
14. Recycling bin is in the pantry.  Just cuz y’all are too lazy to actually walk outside and dump your junk in the recycle bucket out back. Just sayin’. By the way .. organics hit the bucket at the BACK .. if you give a cr@p about that stuff.
15. Yes, I do offer a taxi service at 5am to the AUS airport.  It’s $50,000 per one way.  Your choice, but it’s COB bud.
16. Don’t touch the Henkle knife (knives) .. I have to sacrifice  goat entrails to keep it sharp.  Pretty sure you don’t want any part of that. Just sayin’. You get the steak knives.
17. NEVER turn off the Cranberries. EVER. See rule #3.
18. Whoever gets to the music remote wins. Except when Rule #3 applies. That means Al wins. All the time. Damn dude don’t cry.

Ok, you get the point. Be responsible. Recycle. Use less energy. Don’t be lazy. Book your stay. (It’s only uncomfortable for those of you who don’t and wind up sleeping in the same guest bed .. y’all are NOT sleeping with me).

Oh .. ya, I’m sure you’ll have fun here .. no problems, mate. Yes the lights in the back yard are a secret. DON’T tell Amanda.

• The getting lost was worth the coming home. 

• What I fear, I can create.  We must be willing to let go of the life we planned, so as to have the life that is waiting for us.

Here are 20 things you can do to make your apache configuration more secure.

Disclaimer: The thing about security is that there are no guarantees or absolutes. These suggestions should make your server a bit tighter, but don’t think your server is necessarily secure after following these suggestions.

Additionally some of these suggestions may decrease performance, or cause problems due to your environment. It is up to you to determine if any of the changes I suggest are not compatible with your requirements. In other words proceed at your own risk.

First, make sure you’ve installed latest security patches

There is no sense in putting locks on the windows, if your door is wide open. As such, if you’re not patched up there isn’t really much point in continuing any longer on this list.

Hide the Apache Version number, and other sensitive information.

By default many Apache installations tell the world what version of Apache you’re running, what operating system/version you’re running, and even what Apache Modules are installed on the server. Attackers can use this information to their advantage when performing an attack. It also sends the message that you have left most defaults alone.

There are two directives that you need to add, or edit in your httpd.conf file:

ServerSignature Off
ServerTokens Prod

The ServerSignature appears on the bottom of pages generated by apache such as 404 pages, directory listings, etc.

The ServerTokens directive is used to determine what Apache will put in the Server HTTP response header. By setting it to Prod it sets the HTTP response header as follows:

Server: Apache

If you’re super paranoid you could change this to something other than “Apache” by editing the source code, or by using mod_security (see below).

Make sure apache is running under its own user account and group

Several apache installations have it run as the user nobody. So suppose both Apache, and your mail server were running as nobody an attack through Apache may allow the mail server to also be compromised, and vise versa.

User apache
Group apache

Ensure that files outside the web root are not served

We don’t want apache to be able to access any files out side of its web root. So assuming all your web sites are placed under one directory (we will call this /web), you would set it up as follows:

<Directory />
  Order Deny,Allow
  Deny from all
  Options None
  AllowOverride None
</Directory>
<Directory /web>
  Order Allow,Deny
  Allow from all
</Directory>

Note that because we set Options None and AllowOverride None this will turn off all options and overrides for the server. You now have to add them explicitly for each directory that requires an Option or Override.

Turn off directory browsing

You can do this with an Options directive inside a Directory tag. Set Options to either None or -Indexes

Options -Indexes

Turn off server side includes

This is also done with the Options directive inside a Directory tag. Set Options to either None or -Includes

Options -Includes

Turn off CGI execution

If you’re not using CGI turn it off with the Options directive inside a Directory tag. Set Options to either None or -ExecCGI

Options -ExecCGI

Don’t allow apache to follow symbolic links

This can again can be done using the Options directive inside a Directory tag. Set Options to either None or -FollowSymLinks

Options -FollowSymLinks

Turning off multiple Options

If you want to turn off all Options simply use:

Options None

If you only want to turn off some separate each option with a space in your Options directive:

Options -ExecCGI -FollowSymLinks -Indexes

Turn off support for .htaccess files

This is done in a Directory tag but with the AllowOverride directive. Set it to None.

AllowOverride None

If you require Overrides ensure that they cannot be downloaded, and/or change the name to something other than .htaccess. For example we could change it to .httpdoverride, and block all files that start with .ht from being downloaded as follows:

AccessFileName .httpdoverride
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</Files>

Run mod_security

mod_security is a super handy Apache module written by Ivan Ristic, the author of Apache Security from O’Reilly press.

You can do the following with mod_security:

• Simple filtering
• Regular Expression based filtering
• URL Encoding Validation
• Unicode Encoding Validation
• Auditing
• Null byte attack prevention
• Upload memory limits
• Server identity masking
• Built in Chroot support
• And more

Disable any unnecessary modules

Apache typically comes with several modules installed. Go through the apache module documentation and learn what each module you have enabled actually does. Many times you will find that you don’t need to have the said module enabled.

Look for lines in your httpd.conf that contain LoadModule. To disable the module you can typically just add a # at the beginning of the line. To search for modules run:

grep LoadModule httpd.conf

Here are some modules that are typically enabled but often not needed: mod_imap, mod_include, mod_info, mod_userdir, mod_status, mod_cgi, mod_autoindex.

Make sure only root has read access to apache’s config and binaries

This can be done assuming your apache installation is located at /usr/local/apache as follows:

chown -R root:root /usr/local/apache
chmod -R o-rwx /usr/local/apache

Lower the Timeout value

By default the Timeout directive is set to 300 seconds. You can decrease help mitigate the potential effects of a denial of service attack.

Timeout 45

Limiting large requests

Apache has several directives that allow you to limit the size of a request, this can also be useful for mitigating the effects of a denial of service attack.

A good place to start is the LimitRequestBody directive. This directive is set to unlimited by default. If you are allowing file uploads of no larger than 1MB, you could set this setting to something like:

LimitRequestBody 1048576

If you’re not allowing file uploads you can set it even smaller.

Some other directives to look at are LimitRequestFields, LimitRequestFieldSize and LimitRequestLine. These directives are set to a reasonable defaults for most servers, but you may want to tweak them to best fit your needs. See the documentation for more info.

Limiting the size of an XML Body

If you’re running mod_dav (typically used with subversion) then you may want to limit the max size of an XML request body. The LimitXMLRequestBody directive is only available on Apache 2, and its default value is 1 million bytes (approx 1mb). Many tutorials will have you set this value to 0 which means files of any size may be uploaded, which may be necessary if you’re using WebDAV to upload large files, but if you’re simply using it for source control, you can probably get away with setting an upper bound, such as 10mb:

LimitXMLRequestBody 10485760

Limiting Concurrency

Apache has several configuration settings that can be used to adjust handling of concurrent requests. The MaxClients is the maximum number of child processes that will be created to serve requests. This may be set too high if your server doesn’t have enough memory to handle a large number of concurrent requests.

Other directives such as MaxSpareServers, MaxRequestsPerChild, and on Apache2 ThreadsPerChild, ServerLimit, and MaxSpareThreads are important to adjust to match your operating system, and hardware.

Restricting Access by IP

If you have a resource that should only by accessed by a certain network, or IP address you can enforce this in your apache configuration. For instance if you want to restrict access to your intranet to allow only the 176.16 network:

Order Deny,Allow
Deny from all
Allow from 176.16.0.0/16

Or by IP:

Order Deny,Allow
Deny from all
Allow from 127.0.0.1

Adjusting KeepAlive settings

According to the Apache documentation using HTTP Keep Alive’s can improve client performance by as much as 50%, so be careful before changing these settings, you will be trading performance for a slight denial of service mitigation.

KeepAlive’s are turned on by default and you should leave them on, but you may consider changing the MaxKeepAliveRequests which defaults to 100, and the KeepAliveTimeout which defaults to 15. Analyze your log files to determine the appropriate values.

Run Apache in a Chroot environment

chroot allows you to run a program in its own isolated jail. This prevents a break in on one service from being able to effect anything else on the server.

It can be fairly tricky to set this up using chroot due to library dependencies. I mentioned above that the mod_security module has built in chroot support. It makes the process as simple as adding a mod_security directive to your configuration:

SecChrootDir /chroot/apache

There are however some caveats however, so check out the docs for more info.

Acknowledgments

I have found the book Apache Security to be a highly valuable resource for securing an apache web server. Some of the suggestions listed above were inspired by this book.

Imagine setting out on a journey without a map and signposts. That’s what it would be like if you tried to do your job without feedback from customers, partners, members of your team, and other key stakeholders, said Piau-Phang (PP) Foo, managing director and senior vice president of Global Sales, Asia Pacific and Japan (APJ), in a recent Leading Ideas webcast.

Feedback can be a powerful tool to foster learning and drive better performance. “When executed well and on a consistent basis, it helps get people on track,” said Foo. “It serves as a guide to assist people to know how they are doing and how others perceive their performance.”

Foo cited research that shows that companies that provide frequent feedback energize and motivate their workforce to better performance. They have higher levels of customer satisfaction, hire and retain the best talent, and have better business outcomes.

But giving and receiving feedback, which Foo said is “an objective message about behavior and consequences,” can be challenging. And if you’re like many others, you’ve likely had at least one negative experience when feedback degraded into a verbal wrestling match, an argument about who’s right and wrong.

It doesn’t have to be this way, said Foo. With a little bit of knowledge and preparation, all of us can get better at giving and receiving feedback.

Ten tips for giving feedback

In his webcast, Foo offered HP leaders a range of practical and inspiring ideas for making feedback a competitive advantage, starting with giving feedback:

1. Set expectations.When someone new joins his team, Foo lets that person know that he typically offers prompt feedback. At the same time, he invites the new employee (and everyone else on his team) to give him prompt feedback, as well.
2. Make it informal.Foo tries to make feedback a regular occurrence. “Feedback works best if it is a continual process and not something you do only once or twice a year in a formal session,” he said. “Sometimes, I say to one of my subordinates, ‘Hey, let’s grab a quick lunch so I can give you some feedback.’”
3. Stay focused.Foo says that it is important to focus on just one or two topics at a time—maybe three at the most—so the person receiving feedback is not overwhelmed.
4. Discuss actions, not attributes.People tend to be more open to practical ideas and suggestions that could enhance their job performance than they are to feedback related to aspects of their personality.
5. Be specific.Convey the facts in an objective way, said Foo. For example, describe how an employee’s actions have had an impact on a customer or another member of the team. Avoid expressing emotions and feelings, which can put the other person on the defensive.
6. Check your assumptions.If you plan to give feedback based on something you’ve heard, be sure to investigate the situation for yourself so you can understand the bigger picture and have more empathy. Careless feedback can harm a relationship. “Whatever feedback you give, make sure it’s correct,” Foo said.
7. Be aware of your motivation.People sometimes use “feedback” as a way to get even with or belittle someone. But that’s not true feedback, said Foo. If you are upset about something, take a time out. “Cool down a little bit. Don’t overreact,” he advised.
8. Be balanced.Don’t just focus on the negative. Take a look over a period of time and give specific examples of what the person receiving feedback has done well. Acknowledge his or her contributions to customers and the team.
9. Suggest ways to improve.It’s easy to say that something’s wrong, but the person giving feedback should spend time in advance thinking about ways to improve. “It’s not up to you to come up with all the solutions, but you can start the process,” said Foo.
10. Agree on a time to follow up. Following up can help make feedback stick, but rather than imposing a timeframe, Foo suggests asking the person receiving feedback when he or she would like to talk about the matter again.

Five tips for receiving feedback

Foo also offered practical insights for receiving feedback:

1. Go beyond welcoming feedback; ask for it.If you really want to benefit from feedback, seek it, Foo advised. “Make an effort. It can be as simple as sending a quick email to a colleague and saying, ‘How did I do?’”
2. Manage your emotions.Many of us find it easy to receive feedback when it is positive, but the moment we hear something challenging, we tend to get defensive. “You really need to manage your emotions,” said Foo. “Evaluate the situation before you respond.”
3. Don’t argue, deny, or try to justify.If the feedback you receive catches you by surprise, try to understand the other person’s point of view before you react. Ask for specific examples. For instance, you could say, “When did you see me doing that?”
4. Keep the proper perspective.Feedback usually relates to a specific area of your life, and now you have the opportunity to do something about it. Remember that it’s not about your entire life or you as a person.
5. Take action. After receiving feedback, you have to make a choice: Are you going to act on it, or are you going to ignore it? “I think we have to take action,” said Foo. “If people are willing to give us feedback and we make an effort, it makes an impression.”

Creating a culture of feedback

Feedback can help us learn, grow, and be more fulfilled in our jobs. It can help our team reach higher levels of performance. For these reasons, Foo suggests letting others know that you are open to receiving feedback. Those who might offer you helpful suggestions include people on your team, others in HP, partners and customers.

“Feedback is one of the cheapest, most flexible, yet most powerful tools available to everybody for personal and business success,” said Foo. “It is also perhaps the most underused tool that we have to facilitate learning. I would encourage everybody to use it more often.”

Kihei Boat Launch

Al, Seth, Dani and Andy

Two quiet days in October and the weather was fantastic. Day one I got to dive with AJ and Warren, while the second day I dove with Seth. The fact that every single time I head out with these guys, they have outstanding customer service and attitude .. and that’s not just the awesome sticky buns they consistently provide.

Day one was a great day in the Molokini crater where we saw lots of coral creatures including an extremely large lobster. The second dive at Puu O’Lai had great visibility and lots of turtles and several amazing (apparently rare) fly-bys of four Spotted Eagle Rays.

Spotted Eagle Rays

Bubbles off back wall of Molokini

Day two we hit the back wall of the Molokini Crater with the (literally) breath-taking 350′ expanse of coral and creatures. Dive two on the second day was at Wailea Point with more very friendly turtles.

Thanks again guys – hopefully we’ll see you in another year.

Assumptions:
- webserver root directory is /var/web
- production node is called prod
- development node is called dev
- WordPress database is called wpdb

Procedure to copy production WordPress instance to the development node:
1. Copy webserver www root dir via a tarball
tar czf prod-20110909.tgz /var/web

2. Dump the WordPress database to a MySQL dmp file:
mysqldump -u$mysqluser -p$mysqlpass wpdb | \
 gzip -c > prod-20110909.dmp.gz

3. Copy these two backup files to the dev node:
scp prod-20110909* user@dev:.

On the development node:
4. Unpack the webserver tarball:
mv /var/web /var/web.previous
cd /
tar xzvf prod-20110909.tgz

5. Drop the WordPress database and restore the new version:
mysql> drop database wpdb;
mysql> create database wpdp;
$ gunzip prod-20110909.dmp.gz
$ mysql -u$mysqluser -p wpdb < prod-20110909.dmp

6. Update the WordPress 'siteurl' and 'home' options to point to the development node:
update wp_options set option_value='http://dev.pomeroy.us' where option_name='siteurl';
update wp_options set option_value='http://dev.pomeroy.us' where option_name='home';

Should be all done!

Zest
1134 E. 54th St.
Indianapolis, IN 46220
(317) 466-1853
www.zestexcitingfood.com/

Remembrance of those who lost their lives and those who gave their lives in the line of duty is an important act that we all should honor.

 We will be doing our most difficult cross country mountain bike ride and will give a minute of silence at the top in honor of those who lost their lives as well as in support of the survivors.

Visit www.911day.org and tell the nation what you’ll be doing on 9/11/11.

Update: At 6,398′ on Moose Mountain, we gave a moment of silence.

Now what do we do?

Apply a tried and true multi phase approach .. assess current state, determine desired target state, perform a gap analysis, implement improvements based on priority. Basically we need to establish current state, determine what future state should be, and use the gap analysis as the deliverables of the IT security program. There may be many trade-offs that are made due to limiters like political challenges, funding constraints and difficulty in changing corporate culture. The plan you build with the business gives you the ammunition needed to persuade all your stakeholders of the value in the changes you’ll be proposing.

1. Understand the Current Environment

For a manager or enterprise architect to determine where to start, a current state must be known. This is basically an inventory of what IT security controls, people and processes are in place. This inventory is used to determine what immediately known risks and gaps from relevant security control frameworks exist. The known risks and gaps gives us a starting point to understand where impacts on the business may originate from.

Take the opportunity to socialize foundational security concepts with your new business owners and solicit their input. What are the security related concerns they have? If there has been any articulation of Strengths, Weaknesses, Opportunities, and Threats (SWOT), obtaining that review can also give you an idea of weaknesses or threats that are indicative of missing controls. In the discussions with your new constituents, talk to the infrastructure managers and ask them what security related concerns keep them awake at night – there is likely some awareness but they don’t know how to move forward. Keep in mind most organizations will want a pragmatic approach versus an ivory tower perfect target state.

Some simple questions can quickly give you a picture of the state of security controls. For example, in organizations I’ve worked with, the network administrators could not provide me a complete “layer three” diagram – a diagram that shows all the network segments and how they hang together. It wasn’t that they didn’t want to, the diagrams simply didn’t exist. With over 1,500 network nodes over two data centers and two office complexes, the network group had the topology and configuration “in their heads”. Obvious weaknesses and threats include prevention of succession planning or disaster recovery, poor security transparency, and making nearly any change to the environment higher risk than necessary.

Another example is an organization that had weak asset control. At any point in time it was nearly impossible to determine if unauthorized network nodes existed, since the workstation, notebook, server, virtual machine, switches, firewalls, printers and any other network connected equipment were tracked separately, if at all. No regular audits were performed to reconcile what the organization had purchased was actually what was connected to their networks. This points to weak change control and weak asset control. Without strong asset control, it is difficult to offer assurance to the business owners that serious vulnerabilities have been mitigated to a level they can accept.

Ensure you’re asking questions that will allow you to develop future metrics, such as:

• Do security controls that are in place generate measurable performance statistics?
• How many user accounts are added, disabled, deleted per day/week/month/quarter?
• What volume of inbound email is spam/malware?
• Does the operations team have baselines of normal network, system, application activity?
• Profile of user accounts – how many are inactive (say 90 days)
• How automated is the new hire, dehire, change process? Is there room for manual error?
• How many administrator accounts are there (percentage of all accounts)
• What degree of individual user accountability is there? Are there signed acceptable use agreements?
• Are there accurate network topology and security zone as-built diagrams?
• Is there clear segregation of assets that contain high value data?
• Are content filtering and malware controls deployed?

All these identified issues can then be dropped into a mind map or even a spreadsheet to visualize the highest risks. More on this in a minute.

2. Determine Target State

Next there must be a clear understanding of business goals, since IT (and IT security) always needs traceability back to the goals of a business. Application of good architecture practice then leads to being able to generate a ‘future state’ and a gap analysis. You don’t have time for a full heavy weight analysis, so a good place to start here is to identify what external regulatory compliance mandates exist – there likely are multiples. For example, financial integrity and reporting requirements (SOX / CSOX / JSOX / GLB / Basel / PCI-DSS), privacy (CA-SB-1384, PIPA), critical infrastructure (NERC CIP), government IT systems (FISMA), and health records (HIPAA). Over and above the external regulatory requirements, there may also be a requirement for voluntary or internal compliance mandates, such as corporate policy compliance.

Keep in mind that where a framework like ISO 27002 or NIST SP 800-53 is used, there may be some latitude. For example, your organization may decide that some controls in the framework are not applicable, and exclude them.  So a “C” or “C+” may be as good as your organization wants to get. That’s ok, as long as it’s a concerted decision and you can still hit your mandatory targets.

This is an area where there may a need for substantial effort, so using a common security control framework like ISO 27002 or NIST 800-53 can give a somewhat logical progression of capability maturity that you can build on to close the gaps and start to get a better foundation in place that allows your IT security program to be sustainable. The other benefit to getting to know a good framework is that many compliance mandates are attainable by using the controls in these frameworks – although you may need to adjust a little here or there. The frameworks also act as a checklist for areas to ensure are addressed adequately for your organization.

While using your chosen control framework, also consider a capability maturity level, such as what is outlined in the Carnegie Mellon software Capability Maturity Model (CMM). Where you find deficiencies, choosing controls and configurations that phase in a new control gradually over some predetermined period of time versus expecting a sudden transition from low capability to high capability allows the organization to adapt and culturalize the new controls.

I would suggest using the control framework categories so you can cross reference the issues you’ve uncovered to controls in the framework – you’ll use this to help triage what needs to be focused on first, while helping to capture that data in a way you’ll use for the long term plan.

3. Assess Highest Risks and Identify Operational Wins (gap analysis)

A successful security plan includes executive endorsement of policy, standards, procedures and guidelines. That said, start with the highest risks you identified in step one. Especially the risks that have simple or inexpensive controls that would work. Remember to integrate metrics where possible to enable feedback and improvement. See www.securitymetrics.org for examples of good metrics.

Ensure you are improving:

• Situational awareness (logging, monitoring, reporting and visibility)
• Response capability to Computer Security Incidents
• Long term security controls (build the foundation so you’re not always fighting fires)

There are several assessment methodologies available, including the NSA IAM, Canadian RCMP TRA and Open Source Risk Assessment toolkits and methodologies.

4. Implement Controls to Mitigate Risks

Long term you want to introduce a way to reduce the daily crisis response, by building up the foundational maturity of the organization’s security controls. Short term you need to get some of the major issues and exposures fixed.

Finally changes to process and possibly organization can complete the ability of IT security to stay evergreen (sustainable). Once you’ve made some changes to the environment, it’s important to ensure there is management metrics and process implemented to sustain the changes you’ve made. For instance, if an asset inventory has been done, the value of that goes out the window if there is no measurable, enforceable process to keep the inventory accurate.

Work the evaluation process with your stakeholders and senior management to ensure you are building your roadmap with their support. Laying out a long term plan and showing where you are in the plan will allow you to get increased executive buy-in for more expensive and possibly organizational changes.

You will find that the focus on issues, from whatever source they have been discovered (Architecture Assessments through actual Security Incidents), will tend to be on the highest criticality. This will result in projects or funding for efforts to close a particular issue or exposure – but still results in fire-fighting. You need over the term of your 90 days to ensure you socialize the IT Security Program framework that includes short-term fire-fighting (Urgent) -AND- long term (Foundational) work. This is more than a simple compliance mandate, since companies can be fully compliant with all regulatory mandates, but still get pwned. A rational application of a framework like ISO 27002 or NIST SP 800-53 can result in compliance success and good security.

You will be presented with “stakeholders” that insist on massively grand Total Cost of Ownership (TCO) or foundational Security Policy work, but you must resist such efforts that will undermine and ultimately defeat good security.

This is a plan that I’ve found helpful to survive the first 90 days and possibly build and maintain a positive working relationship with your senior management as they see value in the IT security program. At that point you will have an IT security program that actually works for the organization and is cost effective.

5. Publish a Schedule of Audits

Once some of the foundational controls are in place, gain acceptance and buy-in from your constituents through the use of a well known internal audit schedule. This will help to confirm the effectiveness of steps 1 through 4 and helps in reducing the amount of effort required to perform audits for regulatory mandate compliance proof. In some cases, you may be able to combine these audits, although what you are trying to do here is ensure all the controls you have put into place are functioning as expected, while external audits may not go that deep.

Once you have these steps in place, your security posture will improve and you will experience less fire-drills around suspected incidents.

Technology has evolved since the last MythTV PVR I built, as chronicled here.  Here’s the latest techniques and tech that I’ve used to (start) build(ing) my current PVR. I’ll update this article as I go, as there’s been some bumps along the way, so completion of the project has been slower than I anticipated.

Requirements for my new PVR include:

• Linux operating system for cost and flexibility reasons
• Quiet! Fan-less operation if at all possible, external power supply ok
• Small form factor, black case to fit in with my current home theater gear
• Video capture with MPEG-2 hardware acceleration to help keep the CPU needed as small as possible, in an expansion card format for the most compact physical footprint .. additionally there must be at least two independent tuners
• Analog tuners, but would be good if they were capable of digital for when I eventually move to digital/HD
• IR receiver and transmitter capability for easy remote control and ability of the PVR to use my current set-top box as a source (gives me all the cable company movies and channels that are not available via the basic cable connection
• Ability to schedule at least 10 shows and retain 5 episodes of each show .. also ability to schedule based on show name alone
• Ability to perform post-recording processing, such as removing commercials or changing formats
• Should be able to use a pre-packaged distribution for most if not all of the functions .. I know it’s a home-brew, but I’m tired of messing with individual packages, firmware, and custom codes to make it work. Using a distribution package makes it easier to maintain through updates.
• Want to purchase the parts from the same supplier if possible (ended up using newegg.ca)

Since I already run MythTV, it was an obvious starting point and given I don’t have an affinity to a specific Linux distribution, I looked at Mythbuntu and Mythdora since I’m familiar with and already run both Ubuntu and Fedora distributions.

After downloading the Mythbuntu 10.10 ISO disk image, I discovered I didn’t have my USB DVD drive, so I wanted to create a bootable USB flash disk.  I followed the excellent instructions at https://help.ubuntu.com/community/Installation/FromUSBStick and successfully burned a bootable Mythbuntu disk on a 2GB USB flash disk via a Ubuntu VM running on my MacBook Pro.

The Hardware

The hardware that I chose to use included:

• An Antec ISK-300-65 case, good for fan-less operation
• ASUS AT5IONT-I mainboard dual core Atom D525 CPU
• Hauppauge WinPVR-2250 dual tuner PVR card with MPEG-2 hardware acceleration (PCI-express)
• 4GB DDR3 SO-DIMM memory (2x 2GB)
• 2x 750GB 2.5″ SATA HDDs
• My existing Microsoft MCE USB IR receiver/blaster and remote

I evaluated the very cool and potentially high performance hybrid HDD/SSD disks, but there were too many experiences users expressed that were sub-optimal, most stating the technology is too new. Having a terabyte 2.5″ disk with 4GB of SSD would be sweet, but for now I’m just sticking with 750GB 7200RPM 2.5″ SATA disks. Since I changed my mind and I’m not going to put a DVD drive into the case, I chose to put another HDD in and mirror them up (since there are two SATA adapters on the mainboard and space in the case for two HDD).

The ASUS mainboard is designed for fan-less operations, and coupled with the Antek case as one massive heat sink, it is incredibly quiet. Video outputs are all handled by the mainboard versus the video capture card and include DVI, HDMI and component video outputs. On initial power on, I was somewhat underwhelmed, since although the power on button turned on the blue power light on the mainboard, then spun up the disk and fan, no joy on the mainboard BIOS POST. After some Googling, I found the Asus board uses the very finicky Intel memory controller that is used with the Atom CPU. I purchased a pair of KVR1066D3S7/1G (Kingston 1GB 204-Pin DDR3 SO-DIMM DDR3 1066 (PC3 8500) Laptop Memory) to boot the AT5IONT-I far enough to get the BIOS updated. See the forum thread here for other people’s experiences. Version 312 of the ASUS BIOS did not support the 2GB DIMMs so I was a bit annoyed that I had to purchase 1GB DIMMs (Kingston KVR1066D3S7/1G) in order to get into the BIOS.  I downloaded the 316 BIOS ROM image from the ASUS website and put it onto a FAT formatted USB memory stick, thinking I’d have to go through the pain of booting some form of Windows or DOS to run some lame BIOS updater utility. I was pleasantly surprised to find a BIOS update utility built into the BIOS! All I had to do is plug in the USB stick and select the option to update the BIOS. It worked! Not only the most painless BIOS update I’ve ever done, now the 2GB memory DIMMs work (anyone want to buy my 1GB DIMMs for the cost of shipping?). On to the installation of Mythbuntu.

I originally wanted to have a slim DVD drive to play DVDs but then realized that I don’t even have any movies on DVD any more.  All the oldie goldies that I had, I now have copies in iTunes. Since the mainboard only supports two SATA interfaces, I chose to reserve one for a future redundant HDD (as it turns out I just ordered the extra disk when I purchased the 1GB DIMMs).

The Hauppauge card is a dual-tuner analog/digital that has an IR receiver and blaster – so it can change channels on a cable set top box. The 2250 also has dual tuners so that the conflicts that I often encountered with a single tuner can be avoided. 

OS Install

I tried a couple of All-In-One distributions (Mythdora and Mythbuntu) and even a couple of versions of each.  Seemed like I ran into issues with both distros in different areas. Mythbuntu 10.10 wouldn’t save the Video Sources. Mythdora had a better setup interface than Mythbuntu 10.10, but would not setup a default route for some reason – all the subsequent updates and package installs would obviously fail.  Sigh. Doing a base install of Fedora 14 then installing from ATrpm repositories would go better for the OS install (including full mdadm mirroring of the two SATA drives), but compiling the Hauppauge HVR 2250 analog driver from Steve Toth’s excellent support site would fail with usb_ function call mismatch errors. Apparently the usb_ memory function definitions have changed in recent 2.6 kernels. Arrrg!

Fortunately I set this aside for a while and in the mean time, Mythbuntu came out with release 11.04 … would it work??

So now it works for analog .. exactly what I wanted. Ironically I don’t need the digital tuners for a while yet.

Here’s how:

I downloaded Mythbuntu 11.04 64 bit ISO and created bootable USB flashdrive via
http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3 on my HP notebook (Windows 7). Booted off the USB and selected the Install option. Ultimately I wanted to partition the drives and use mdadm software RAID 1 with LVM2 on top for partition and filesystem management options. No matter how I tried, the Mythbuntu 11.04 installer just would not let me do an install in that configuration. So I did a vanilla install, configured things the way I wanted THEN did a transition to LVM2 mirroring setup.

1. Use USB stick to boot Mythbuntu 11.04 and perform MythTV install
Use the following partition table on /dev/sda with all primary partitions and ignore /dev/sdb for now – note you’ll need to use /srv for the MythTV storage fs as those are the mount point options available in the install image. Don’t worry, we’ll change it later to /storage and only /dev/sda1 (/boot) will remain after we’re done the conversion to LVM mirroring.
/dev/sda1 /boot 150MB
/dev/sda2 / 8GB
/dev/sda3 swap 8GB
/dev/sda4 /srv 630GB

Once the install is done, change the /srv filesystem to /storage and make it owned by user mythtv then create the storage directories that MythTV will use for LiveTV and Recordings.
umount /srv
vi /etc/fstab (change /srv to /storage)
mkdir /storage
mount /storage
mkdir /storage/livetv
mkdir /storage/recordings
mkdir /storage/db_backups
chown mythtv:mythtv /storage/*

2. Compile V4L code
Install tools needed to get and build the code
apt-get install git patch patchutils libproc-processtable-perl gcc make

Get the code base and build it – based on the forum post “How to Obtain, Build and Install V4L-DVB Device Drivers”
git clone git://linuxtv.org/media_build.git
cd media_build
./build
sudo make install


3. Download HVR 2250 firmware and install in /lib/firmware
wget http://www.steventoth.net/linux/hvr22xx/firmwares/4019072/NXP7164-2010-03-10.1.fw
sudo cp NXP7164-2010-03-10.1.fw /lib/firmware

I found that the HVR 2250 card wasn’t completely recognized even with the firmware in place, as seen in dmesg output .. and no /dev/video* or /dev/dvb/… devices were created. Googling found a few forum posts that discuss the issue .. the solution: create /etc/modprobe.d/saa7164.conf and force a card selection.

Jun 23 22:17:54 pvr kernel: [10.642158] saa7164 driver loaded
Jun 23 22:17:54 pvr kernel: [10.642321] saa7164 0000:07:00.0: PCI INT A -> GSI 19 (level, low) ...
Jun 23 22:17:54 pvr kernel: [10.643371] saa7164[0]: Your board isn't known (yet) to the driver.
Jun 23 22:17:54 pvr kernel: [10.643376] saa7164[0]: Try to pick one of the existing card configs via
Jun 23 22:17:54 pvr kernel: [10.643380] saa7164[0]: card=<n> insmod option.  Updating to the latest
Jun 23 22:17:54 pvr kernel: [10.643384] saa7164[0]: version might help as well.
Jun 23 22:17:54 pvr kernel: [10.643395] saa7164[0]: Here are valid choices for the card=<n> insmod option:
Jun 23 22:17:54 pvr kernel: [10.643403] saa7164[0]:    card=0 -> Unknown
Jun 23 22:17:54 pvr kernel: [10.643410] saa7164[0]:    card=1 -> Generic Rev2
Jun 23 22:17:54 pvr kernel: [10.643417] saa7164[0]:    card=2 -> Generic Rev3
Jun 23 22:17:54 pvr kernel: [10.643424] saa7164[0]:    card=3 -> Hauppauge WinTV-HVR2250

To set a card number option, create a modprobe directive file /etc/modprobe.d/saa7164.conf
options saa7164 card=3

Now reboot and watch the dmesg output to ensure the firmware is loaded properly
Aug 18 19:31:29 pvr1 kernel: [24.480644] saa7164 driver loaded
Aug 18 19:31:29 pvr1 kernel: [24.480891] saa7164 0000:07:00.0: PCI INT A -> GSI 19 (level, low) ...
Aug 18 19:31:29 pvr1 kernel: [24.490973] CORE saa7164[0]: subsystem: 0070:8891, board: Hauppauge WinTV-HVR2250 [card=3,insmod option]
Aug 18 19:31:29 pvr1 kernel: [24.490992] saa7164[0]/0: found at 0000:07:00.0, rev: 129, irq: 19, latency: 0, mmio: 0xfb800000
Aug 18 19:31:29 pvr1 kernel: [24.700362] saa7164_downloadfirmware() Waiting for firmware upload (NXP7164-2010-03-10.1.fw)
Aug 18 19:31:29 pvr1 kernel: [27.153217] saa7164_downloadfirmware() firmware read 4019072 bytes.
Aug 18 19:31:29 pvr1 kernel: [27.153227] saa7164_downloadfirmware() firmware loaded.
Aug 18 19:31:29 pvr1 kernel: [27.153257] saa7164_downloadfirmware() SecBootLoader.FileSize = 4019072
Aug 18 19:31:29 pvr1 kernel: [27.153269] saa7164_downloadfirmware() FirmwareSize = 0x1fd6
Aug 18 19:31:29 pvr1 kernel: [27.153276] saa7164_downloadfirmware() BSLSize = 0x0
Aug 18 19:31:29 pvr1 kernel: [27.153282] saa7164_downloadfirmware() Reserved = 0x0
Aug 18 19:31:29 pvr1 kernel: [27.153289] saa7164_downloadfirmware() Version = 0x1661c00
Aug 18 19:31:29 pvr1 kernel: [27.304006] Modules linked in: nvidia(P+) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel(+) ir_lirc_codec lirc_dev ir_mce_kbd_decoder ir_sony_decoder snd_hda_codec ir_jvc_decoder snd_hwdep snd_pcm snd_seq_midi rc_rc6_mce psmouse ir_rc6_decoder ir_rc5_decoder mceusb serio_raw snd_rawmidi ir_nec_decoder joydev snd_seq_midi_event rc_core snd_seq saa7164(+) snd_timer snd_seq_device snd xhci_hcd asus_atk0110 dvb_core v4l2_common videodev media v4l2_compat_ioctl32 tveeprom soundcore snd_page_alloc lp parport usbhid hid ahci r8169 libahci
Aug 18 19:31:35 pvr1 kernel: [34.380040] saa7164_downloadimage() Image downloaded, booting...
Aug 18 19:31:35 pvr1 kernel: [34.490037] saa7164_downloadimage() Image booted successfully.
Aug 18 19:31:36 pvr1 kernel: [36.830033] saa7164_downloadimage() Image downloaded, booting...
Aug 18 19:31:38 pvr1 kernel: [38.270702] saa7164_downloadimage() Image booted successfully.
Aug 18 19:31:38 pvr1 kernel: [38.996628] DVB: registering new adapter (saa7164)
Aug 18 19:31:38 pvr1 kernel: [38.999488] saa7164[0]: registered device video0 [mpeg]
Aug 18 19:31:39 pvr1 kernel: [39.231624] saa7164[0]: registered device video1 [mpeg]
Aug 18 19:31:39 pvr1 kernel: [39.443888] saa7164[0]: registered device vbi0 [vbi]
Aug 18 19:31:39 pvr1 kernel: [39.444038] saa7164[0]: registered device vbi1 [vbi]


4. Run MythTV Backend setup, use IVTV MPEG driver and /dev/video0, /dev/video1

This configuration will result in the NTSC (analog) tuners to function with MythTV by initializing the tuner as an “IVTV MPEG-2 Encoder” defined for /dev/video0 and /dev/video1.

<I’ll post an update here for each of the backend setup steps, but for now follow the setup steps in order starting with General>

Under Storage, change the default directories for Recordings, LiveTV and Database Backups to the directories you created in step 1.

5. Setup mail transport agent so we can send status email
apt-get install postfix bsd-mailx

Since I locate the PVR behind a firewall on a residential Internet connection, I choose “Internet connected with Smart (Relay) host”
shawmail.cg.shawcable.net

6. Fix up some of the annoying outstanding problems

Arrow Buttons Repeat

For some reason, the Windows Media Center remote control IR codes are working but arrow button presses cause double movements. Huh? Ok, Google solves it:

MCE remote menu entries skipping twice for every button push
http://www.mythtv.org/wiki/MCE_Remote#Arrow_Buttons_Repeat

When navigating the menus in MythTV, it may appear that each time you hit the up or down arrow, that the button his hit twice but if you are using irw, the button only appears to be pressed once. This is likely caused by another kernel module that is attempting to treat the MCE remote as a keyboard. As a test try unloading a few modules
modprobe -r ir_rc6_decoder
modprobe -r rc_rc6_mce
modprobe -r ir_rc5_decoder

Since this solved the problem, the post author suggests the following command on system boot (in /etc/rc.local)
echo lirc > /sys/class/rc/rc0/protocols

Audio Pauses and Stutters

Turns out the default 4096 buffer size is too small for this system, so I increase it to 16384 (trial and error).

Put these lines in /etc/rc.local:

# prevent Arrow Button repeat
echo lirc > /sys/class/rc/rc0/protocols
# increase of audio buffer - from 4096 default to 16384
echo 16384 | tee /proc/asound/card0/pcm1p/sub0/prealloc

7. Setup the Disk Mirroring

I’m not going to use mdadm (dang it!) because mdadm (3.1.4) breaks initramfs (forum posts here). People upgrading Ubuntu to the “natty” release experienced this behavior where initramfs could not mount /root. For now I’ll use LVM2 to do the mirroring of

/
swap
/storage

but /boot will still be a standalone ext4 partition and filesystem on /dev/sda1. For recovery if /dev/sda dies, I’ll partition /dev/sdb the same and keep /dev/sdb1 in sync with /dev/sda1 – as well as have grub install a boot loader on both /dev/sda and /dev/sdb.  With the Asus AT5IONT-I mainboard, you can designate which SATA disk is the “Primary” and which is the “Secondary”. Worst case, if /dev/sda dies the Secondary drive can be manually mapped to the Primary disk (/dev/sda). Whew. A lot of extra work because mdadm is broken!

7.1. Install the lvm2 package

apt-get install lvm2

7.2. Partition the second drive with the desired end state

fdisk /dev/sdb

primary partition 1, 150M, set active, partition type flags “83″ (normal Linux fs)
primary partition 2, rest of the disk, partition type flags “fd” (Linux logical volume)

7.3. Setup the first LVM partition for pvr on /dev/sdb

# initialize the LVM volume
pvcreate /dev/sdb2
pvdisplay /dev/sdb2
# create the volume group
vgcreate rootvg /dev/sdb2
# create the logical volumes (with extents from one physical disk)
lvcreate -L 8G -n lv_root rootvg
lvcreate -L 8G -n lv_swap rootvg
lvcreate -L 650G -n lv_storage rootvg

This gives us the volumes we’ll use in our final configuration. Go ahead a get the filesystem contents copied across to the new LVM volumes.

# create the filesystems
mkfs.ext4 /dev/rootvg/lv_root
mkfs.ext4 /dev/rootvg/lv_storage
mkswap /dev/rootvg/lv_swap
# copy the old fs to new fs
mkdir /mnt/root
mount /dev/rootvg/lv_root /mnt/root
cd /
find . -xdev -print | cpio -pmd /mnt/root
umount /mnt/root
mkdir /mnt/storage
mount /dev/rootvg/lv_storage /mnt/storage
cd /storage
find . -xdev -print | cpio -pmd /mnt/storage
umount /mnt/storage

7.4. Update boot configuration to use LVM root volume

Copy the /dev/sda1 /boot filesystem to /dev/sdb1 for a backup in case something goes horribly wrong, you’ll at least have a starting point to recover.

To update GRUB to use the LVM device, add GRUB_DEVICE=/dev/mapper/rootvg-lv_root
to /etc/default/grub and disable the UUID volume label tracking by uncommenting the GRUB_DISABLE_LINUX_UUID=true line.

Update the /boot/grub/grub.cfg by running
cp /boot/grub/grub.cfg /boot/grub/grub.cfg.orig
update-grub -o /boot/grub/grub.cfg
You should see update-grub detect the original boot env on /dev/sda1 (/boot) with a root of /dev/sda2 (where we configured /) and you should also see it detect the new root environment on /dev/mapper/rootvg-lv_root.

Then install the grub boot environment on /dev/sda and optionally /dev/sdb. Note /dev/sdb will not have a boot block or env loaded yet, so no worries about having to save what might be there.
grub-install /dev/sda
grub-install /dev/sdb

Update the /etc/fstab to swing the filesystems over to the LVM volumes – update the device specs from their UUID labels to /dev/mapper/rootvg-lv_root, /dev/mapper/rootvg-lv_swap and /dev/mapper/rootvg-lv_storage
cp /etc/fstab /etc/fstab.orig
vi /etc/fstab

It should look something like
# / was on /dev/sda2 during installation
#UUID=fc0fa1e9-e2b6-4d11-9a51-d3c432bb3137 / ext4 errors=remount-ro 0 1
/dev/mapper/rootvg-lv_root / ext4 errors=remount-ro 0 1
# leave /boot alone
/dev/sda1 /boot ext4 defaults 0 2
# /storage was on /dev/sda4 during installation
#UUID=131514d2-3911-45df-8d6f-b9a19f2379bb /storage ext4 defaults 0 2
/dev/mapper/rootvg-lv_storage /storage ext4 errors=remount-ro 2
# swap was on /dev/sda3 during installation
#UUID=eedc9a3b-d957-4904-988e-32b117def5ac none swap sw 0 0
/dev/mapper/rootvg-lv_swap none swap sw 0 0

This is the nail-biting time, now reboot. When GRUB comes up and shows you the boot environments, select the normal boot Ubuntu Linux with the root on /dev/mapper/rootvg-lv_root.

7.5. Extend the LVM to use /dev/sda
Now we’re running on the LVM volumes on /dev/sdb, we want to reclaim the plain 0×83 Linux filesystem partitions off /dev/sda and add them to the rootvg volume group then extend each logical volume so it has a mirror on /dev/sda.

# delete /dev/sda2, /dev/sda3, /dev/sda4 partitions
# add /dev/sda2 as the remaining disk, toggle the partition
# type flags to 0xfd (Linux LVM)
fdisk /dev/sda

Now clear the first few blocks of /dev/sda2 since it will still have a Linux ext4 filesystem signature on it and we don’t want to confuse LVM.
dd if=/dev/zero of=/dev/sda2 count=100

Extend the volume group to include /dev/sda2 and add a mirror onto each logical volume.

# initialize the /dev/sda2 partition for LVM
pvcreate /dev/sda2
pvdisplay /dev/sda2
# extend the volume group to include /dev/sda2
vgextend rootvg /dev/sda2
vgdisplay -v
# now extend each logical volume to /dev/sda2
lvconvert -m1 --mirrorlog core /dev/rootvg/lv_root /dev/sda2
lvconvert -m1 --mirrorlog core /dev/rootvg/lv_swap /dev/sda2
lvconvert -m1 --mirrorlog core /dev/rootvg/lv_storage /dev/sda2

Let this run for a while, the system will be very busy syncing (re-silvering) the physical extents on /dev/sdb2 to /dev/sda2.

Side note: If something messes up and you need to remove the /dev/sda2 or /dev/sdb2 volume, or if you need to tear down the lvm2 setup (such as to remove or play around with RAID volumes), use

lvremove /dev/mapper/rootvg-lv_root
vgremove rootvg
pvremove /dev/sdb2

so you don’t have problems with residual signatures when you try to initialize the LVM volume group an physical devices again.

7.6. Update /boot on /dev/sdb

mount /dev/sdb1 /mnt/boot
cd /boot
find . -print | cpio -pvmd /mnt/boot
umount /mnt/boot

Now you should be able to boot off either /dev/sda or /dev/sdb.

8. Remaining issues
Despite turning off the screen saver, the HDMI to TV output dims occasionally and I’m pretty sure it’s not my Sony Bravia that’s doing it.

Audio is still a pain in the butt – I’m using the analog audio out on the Asus mainboard into a pair of Audio Engine speakers to get audio, since I think the HDMI driver needs to be changed to support audio over HDMI. I’m using the Open Source video driver instead of the nVidia and I think that’s the culprit. No time to test it right now though.

Update:
Tried out a LVM volume extend for the /storage filesystem:

df -h /storage
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_storage
669G 70G 565G 11% /storage

vgdisplay
— Volume group —
VG Name rootvg
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 11
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 2
Act PV 2
VG Size 1.36 TiB
PE Size 4.00 MiB
Total PE 357628
Alloc PE / Size 355728 / 1.36 TiB
Free PE / Size 1900 / 7.42 GiB
VG UUID JwB28k-Eeg6-HNq0-Ghdn-r4db-mNqd-fZWYfG

Since vgdisplay shows we have free space (7GB), issue the lvextend command:
lvextend -L +1G /dev/rootvg/lv_storage
Extending 2 mirror images.
Extending logical volume lv_storage to 679.78 GiB
Logical volume lv_storage successfully resized

After extending the logical volume, we can extend the filesystem. We will extend the fs while it’s mounted, since the current versions of resize2fs allow online extension or shrinkage.

resize2fs /dev/mapper/rootvg-lv_storage
resize2fs 1.41.14 (22-Dec-2010)
Filesystem at /dev/mapper/rootvg-lv_storage is mounted on /storage; on-line resizing required
old desc_blocks = 43, new_desc_blocks = 43
Performing an on-line resize of /dev/mapper/rootvg-lv_storage to 178200576 (4k) blocks.
The filesystem on /dev/mapper/rootvg-lv_storage is now 178200576 blocks long.

Yay! LVM sure makes fs and volume manipulation easy.

df -h /storage
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_storage
670G 70G 566G 11% /storage